[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:12 dist-test-slave-fzwk krb5kdc[27928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:12 dist-test-slave-fzwk krb5kdc[27928](info): set up 2 sockets
Sep 29 03:22:12 dist-test-slave-fzwk krb5kdc[27928](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:22:13.296283 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:13.422164 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44637
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:13.504405 27941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:13.504643 27941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:13.504667 27941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:13.505910 27941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:13.505962 27941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:13.505977 27941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:13.505990 27941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:13.506034 27941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:13.507668 27941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:13.507951 27941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:13.508214 27941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:13.513581 27946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:13.513608 27947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:13.513795 27949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:13.513809 27941 server_base.cc:1034] running on GCE node
I20260929 03:22:13.514075 27941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:13.514417 27941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:13.515584 27941 hybrid_clock.cc:648] HybridClock initialized: now 1790652133515576 us; error 95 us; skew 500 ppm
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:13.518309 27941 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:13.518843 27941 webserver.cc:460] Webserver started at http://127.27.68.126:42117/ using document root <none> and password file <none>
I20260929 03:22:13.519109 27941 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:13.519186 27941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:13.519399 27941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:13.520571 27941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "7fb1b653cb2a4c6180dc653875611f7e"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "0e1646cbca69e1fea794cab37a4db169"
server_key_iv: "a7b77072e5a0df0244e7c8ce2e2d624c"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.520960 27941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "7fb1b653cb2a4c6180dc653875611f7e"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "0e1646cbca69e1fea794cab37a4db169"
server_key_iv: "a7b77072e5a0df0244e7c8ce2e2d624c"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.522526 27941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:13.523341 27956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.523526 27941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:13.523636 27941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "7fb1b653cb2a4c6180dc653875611f7e"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "0e1646cbca69e1fea794cab37a4db169"
server_key_iv: "a7b77072e5a0df0244e7c8ce2e2d624c"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.523728 27941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:13.550191 27941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:13.552838 27941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:13.553030 27941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:13.557143 27941 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44637
I20260929 03:22:13.557206 28008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44637 every 8 connection(s)
I20260929 03:22:13.557606 27941 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:13.558530 28009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:13.560945 28009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e: Bootstrap starting.
I20260929 03:22:13.561570 28009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:13.561872 28009 log.cc:824] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:13.562691 28009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e: No bootstrap required, opened a new log
I20260929 03:22:13.564371 28009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } }
I20260929 03:22:13.564524 28009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:13.564548 28009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb1b653cb2a4c6180dc653875611f7e, State: Initialized, Role: FOLLOWER
I20260929 03:22:13.564733 28009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } }
I20260929 03:22:13.564798 28009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:13.564841 28009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:13.564898 28009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:13.565498 28009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } }
I20260929 03:22:13.565615 28009 leader_election.cc:302] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fb1b653cb2a4c6180dc653875611f7e; no voters:
I20260929 03:22:13.565820 28009 leader_election.cc:288] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:13.565863 28012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:13.566054 28012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [term 1 LEADER]: Becoming Leader. State: Replica: 7fb1b653cb2a4c6180dc653875611f7e, State: Running, Role: LEADER
I20260929 03:22:13.566108 28009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:13.566149 28012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } }
I20260929 03:22:13.566582 28013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fb1b653cb2a4c6180dc653875611f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } } }
I20260929 03:22:13.566665 28013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:13.566748 28014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fb1b653cb2a4c6180dc653875611f7e. Latest consensus state: current_term: 1 leader_uuid: "7fb1b653cb2a4c6180dc653875611f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fb1b653cb2a4c6180dc653875611f7e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44637 } } }
I20260929 03:22:13.566831 28014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:13.566882 28020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:13.567421 28020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:13.567653 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 27941
I20260929 03:22:13.567763 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:13.568548 27921 external_mini_cluster.cc:1428] Setting key 243c6ce1e043cbd48dbee09950679b43
I20260929 03:22:13.569176 28020 catalog_manager.cc:1347] Generated new cluster ID: 55f016c21dfa444eb2fbde677046c665
I20260929 03:22:13.569233 28020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:13.584316 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:13.569848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.27.68.126:44637)
0929 03:22:13.570498 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:13.570508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:13.570544 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:22:13.572109 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:13.572115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:13.572173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:13.572335 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:13.573860 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.574867 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.576006 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.576471 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.576594 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:13.580434 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:13.580439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:13.580440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:13.580461 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:13.582208 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.582684 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.582687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.582687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.582779 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.583062 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.583063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.583063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.583405 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:13.583590 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:13.583869 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:22:13.584007 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":608,"thread_start_us":85,"threads_started":1}
I20260929 03:22:13.585907 28020 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:13.586405 28020 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:13.591262 28020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fb1b653cb2a4c6180dc653875611f7e: Generated new TSK 0
I20260929 03:22:13.591456 28020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:13.626974 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:13.716099 28035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:13.716334 28035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:13.716359 28035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:13.717494 28035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:13.717538 28035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:13.717566 28035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:13.718961 28035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:13.719254 28035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:13.719547 28035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:22:13.725076 28035 server_base.cc:1034] running on GCE node
W20260929 03:22:13.725059 28040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:13.725059 28041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:13.725050 28043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:13.725433 28035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:13.725739 28035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:13.726913 28035 hybrid_clock.cc:648] HybridClock initialized: now 1790652133726902 us; error 57 us; skew 500 ppm
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:13.729635 28035 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:13.730149 28035 webserver.cc:460] Webserver started at http://127.27.68.65:42031/ using document root <none> and password file <none>
I20260929 03:22:13.730403 28035 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:13.730475 28035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:13.730634 28035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:13.731879 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "c3b562bb86e042ce91c622ff62652473"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "03b80b93b0b6704b1ed58fbc17b7ec25"
server_key_iv: "acfec8535910d8c6bbfae5db4fb63bce"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.732307 28035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "c3b562bb86e042ce91c622ff62652473"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "03b80b93b0b6704b1ed58fbc17b7ec25"
server_key_iv: "acfec8535910d8c6bbfae5db4fb63bce"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.733865 28035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:13.734637 28050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.734817 28035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:13.734923 28035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "c3b562bb86e042ce91c622ff62652473"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "03b80b93b0b6704b1ed58fbc17b7ec25"
server_key_iv: "acfec8535910d8c6bbfae5db4fb63bce"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.734999 28035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:13.747201 28035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:13.749245 28035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:13.749423 28035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:13.749675 28035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:13.750005 28035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:13.750063 28035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.750105 28035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:13.750134 28035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.756170 28035 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:45431
I20260929 03:22:13.756213 28163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:45431 every 8 connection(s)
I20260929 03:22:13.756589 28035 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:13.762697 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28035
I20260929 03:22:13.762805 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:13.762962 27921 external_mini_cluster.cc:1428] Setting key 299221b99a9c5a6134ffa5963d9dc60f
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:13.766991 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:13.757335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:48261 (local address 127.27.68.126:44637)
0929 03:22:13.757382 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:13.757384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:13.757638 (+ 254us) server_negotiation.cc:407] Connection header received
0929 03:22:13.758223 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:13.758224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:13.758236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:13.758283 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:13.759446 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.759957 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.760998 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.761204 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.761315 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:13.763757 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:13.763759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:13.763759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:13.763768 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:13.765289 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.765842 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.765843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.765843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.765901 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.766180 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.766181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.766181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.766309 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:13.766387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:13.766788 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 03:22:13.766888 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:13.767254 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:13.757329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44637 (local address 127.27.68.65:48261)
0929 03:22:13.757516 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:13.757553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:13.758111 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:13.758378 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:13.758381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:13.759071 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:13.759345 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:13.759352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.760057 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.760058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:13.760896 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:13.760899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.761327 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.761328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:13.761463 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:13.761973 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:13.761981 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:13.763653 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:13.765380 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:13.765389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:13.765392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:13.765759 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:13.765974 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:13.765975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:13.765976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:13.766108 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:13.766378 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:13.766385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:13.766622 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 03:22:13.766751 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:22:13.766969 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260929 03:22:13.767745 28164 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44637
I20260929 03:22:13.767843 28164 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:13.768061 28164 heartbeater.cc:511] Master 127.27.68.126:44637 requested a full tablet report, sending...
I20260929 03:22:13.768599 27973 ts_manager.cc:194] Registered new tserver with Master: c3b562bb86e042ce91c622ff62652473 (127.27.68.65:45431)
I20260929 03:22:13.769227 27973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:48261
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:13.790416 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:13.873872 28169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:13.874091 28169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:13.874116 28169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:13.875252 28169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:13.875291 28169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:13.875319 28169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:13.876730 28169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:13.876930 28169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:13.877108 28169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:13.882098 28177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:13.882189 28169 server_base.cc:1034] running on GCE node
W20260929 03:22:13.882103 28174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:13.882098 28175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:13.882694 28169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:13.882900 28169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:13.884087 28169 hybrid_clock.cc:648] HybridClock initialized: now 1790652133884070 us; error 28 us; skew 500 ppm
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:13.886636 28169 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:13.887050 28169 webserver.cc:460] Webserver started at http://127.27.68.66:34779/ using document root <none> and password file <none>
I20260929 03:22:13.887331 28169 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:13.887380 28169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:13.887475 28169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:13.888479 28169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "2f93948e32c54dbbb8dbe0e05bf2efc8"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "2325b02122e6277c62f9ddc47fb75222"
server_key_iv: "90e97ffc0684028e1d1cbafea6b49ac6"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.888783 28169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "2f93948e32c54dbbb8dbe0e05bf2efc8"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "2325b02122e6277c62f9ddc47fb75222"
server_key_iv: "90e97ffc0684028e1d1cbafea6b49ac6"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.890151 28169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:13.890813 28184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.890983 28169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:13.891076 28169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "2f93948e32c54dbbb8dbe0e05bf2efc8"
format_stamp: "Formatted at 2026-09-29 03:22:13 on dist-test-slave-fzwk"
server_key: "2325b02122e6277c62f9ddc47fb75222"
server_key_iv: "90e97ffc0684028e1d1cbafea6b49ac6"
server_key_version: "encryptionkey@0"
I20260929 03:22:13.891168 28169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:13.937099 28169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:13.939585 28169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:13.939738 28169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:13.939983 28169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:13.940323 28169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:13.940377 28169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.940423 28169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:13.940451 28169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:13.946193 28169 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42707
I20260929 03:22:13.946251 28297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42707 every 8 connection(s)
I20260929 03:22:13.946581 28169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:22:13 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:13.955592 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:13.947339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:45179 (local address 127.27.68.126:44637)
0929 03:22:13.947406 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:13.947408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:13.947612 (+ 204us) server_negotiation.cc:407] Connection header received
0929 03:22:13.948072 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:13.948073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:13.948083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:13.948135 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:13.949064 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.949553 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.950526 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.950707 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.950764 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:13.952932 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:13.952934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:13.952934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:13.952941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:13.954346 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.954738 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.954739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.954739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.954791 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:13.955001 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:13.955001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:13.955002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:13.955105 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:13.955154 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:13.955450 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:22:13.955523 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:13.955804 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:13.947345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44637 (local address 127.27.68.66:45179)
0929 03:22:13.947488 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:13.947525 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:13.947990 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:13.948199 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:13.948202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:13.948781 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:13.948984 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:13.948989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.949632 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.949633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:13.950452 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:13.950454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:13.950783 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:13.950783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:13.950887 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:13.951450 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:13.951459 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:13.952839 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:13.954423 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:13.954429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:13.954431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:13.954665 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:13.954849 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:13.954849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:13.954849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:13.954948 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:13.955160 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:13.955164 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:13.955335 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 03:22:13.955423 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:22:13.955547 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260929 03:22:13.956383 28298 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44637
I20260929 03:22:13.956405 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28169
I20260929 03:22:13.956485 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:13.956498 28298 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:13.956668 27921 external_mini_cluster.cc:1428] Setting key 090f9a0b08cc0d5648d3f7ee559d7808
I20260929 03:22:13.956708 28298 heartbeater.cc:511] Master 127.27.68.126:44637 requested a full tablet report, sending...
I20260929 03:22:13.957083 27972 ts_manager.cc:194] Registered new tserver with Master: 2f93948e32c54dbbb8dbe0e05bf2efc8 (127.27.68.66:42707)
I20260929 03:22:13.957484 27972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:45179
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:13.980283 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:14.062804 28303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:14.063063 28303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:14.063104 28303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:14.064343 28303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:14.064424 28303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:14.064473 28303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:14.066074 28303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44637
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:14.066483 28303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:14.066804 28303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:14.071907 28311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:14.071899 28309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:14.071904 28308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:14.072059 28303 server_base.cc:1034] running on GCE node
I20260929 03:22:14.072348 28303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:14.072609 28303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:14.073784 28303 hybrid_clock.cc:648] HybridClock initialized: now 1790652134073775 us; error 35 us; skew 500 ppm
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:14.076431 28303 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:14.076913 28303 webserver.cc:460] Webserver started at http://127.27.68.67:40083/ using document root <none> and password file <none>
I20260929 03:22:14.077103 28303 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:14.077143 28303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:14.077236 28303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:14.078248 28303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "4d802afe46304945b5dd67ab9917d0fb"
format_stamp: "Formatted at 2026-09-29 03:22:14 on dist-test-slave-fzwk"
server_key: "2840edec562f5fe3b946562e5d88f904"
server_key_iv: "a8baf6239968ed7d8a65680795c17a0c"
server_key_version: "encryptionkey@0"
I20260929 03:22:14.078624 28303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "4d802afe46304945b5dd67ab9917d0fb"
format_stamp: "Formatted at 2026-09-29 03:22:14 on dist-test-slave-fzwk"
server_key: "2840edec562f5fe3b946562e5d88f904"
server_key_iv: "a8baf6239968ed7d8a65680795c17a0c"
server_key_version: "encryptionkey@0"
I20260929 03:22:14.080032 28303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:14.080760 28318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:14.080947 28303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:14.081032 28303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "4d802afe46304945b5dd67ab9917d0fb"
format_stamp: "Formatted at 2026-09-29 03:22:14 on dist-test-slave-fzwk"
server_key: "2840edec562f5fe3b946562e5d88f904"
server_key_iv: "a8baf6239968ed7d8a65680795c17a0c"
server_key_version: "encryptionkey@0"
I20260929 03:22:14.081086 28303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:14.092341 28303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:14.094308 28303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:14.094420 28303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:14.094605 28303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:14.094915 28303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:14.094947 28303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:14.094991 28303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:14.095017 28303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:14.100749 28303 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:43833
I20260929 03:22:14.100819 28431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:43833 every 8 connection(s)
I20260929 03:22:14.101238 28303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:14.104952 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28303
I20260929 03:22:14.105047 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:14.105217 27921 external_mini_cluster.cc:1428] Setting key 026ac7c67c0575c9936c7c0477a2d32e
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:14.110929 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:14.101978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:37677 (local address 127.27.68.126:44637)
0929 03:22:14.102045 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:14.102047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:14.102254 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:22:14.102780 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:14.102781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:14.102793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:14.102852 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:14.103888 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.104392 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.105518 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.105798 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.105883 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:14.108123 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:14.108125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:14.108125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:14.108133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:14.109600 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.110040 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.110042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.110042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.110108 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.110374 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.110374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.110375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.110489 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:14.110541 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:14.110793 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 03:22:14.110863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:22:14.111158 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:14.101963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44637 (local address 127.27.68.67:37677)
0929 03:22:14.102130 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:14.102167 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:14.102673 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:14.102926 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:14.102930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:14.103578 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:14.103780 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:14.103785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.104488 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.104489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:14.105434 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:14.105436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.105892 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.105894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:14.106032 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:14.106628 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:14.106637 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:14.108020 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:14.109709 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:14.109718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:14.109720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:14.109960 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:14.110186 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:14.110187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:14.110188 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:14.110305 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:14.110546 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:14.110550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:14.110679 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 03:22:14.110765 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:22:14.110894 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260929 03:22:14.111640 28432 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44637
I20260929 03:22:14.111723 28432 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:14.111914 28432 heartbeater.cc:511] Master 127.27.68.126:44637 requested a full tablet report, sending...
I20260929 03:22:14.112322 27972 ts_manager.cc:194] Registered new tserver with Master: 4d802afe46304945b5dd67ab9917d0fb (127.27.68.67:43833)
I20260929 03:22:14.112710 27972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:37677
I20260929 03:22:14.117180 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:14.140401 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:14.132536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.27.68.65:45431)
0929 03:22:14.132733 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:14.132738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:14.132749 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:14.132777 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:14.132779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:14.132792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:14.132858 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:14.133425 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.134141 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.134882 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.135199 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.135312 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:14.137575 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:14.137584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:14.137584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:14.137591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:14.139329 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.139661 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.139664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.139665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.139733 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.139896 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.139897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.139897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.140030 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:14.140145 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:14.140207 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:22:14.140272 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
W20260929 03:22:14.140791 28078 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38568
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[27928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:14.163323 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:14.156613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.27.68.65:45431)
0929 03:22:14.156695 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:14.156697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:14.156750 (+ 53us) server_negotiation.cc:407] Connection header received
0929 03:22:14.156801 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:14.156802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:14.156814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:14.156866 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:14.157422 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.157908 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.158590 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:14.158775 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:14.158824 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:14.161119 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:14.161121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:14.161121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:14.161128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:14.162339 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.162646 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.162646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.162647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.162695 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:14.162855 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:14.162855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:14.162856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:14.162947 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:14.163030 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:14.163102 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:22:14.163166 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:22:14.163933 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28035
I20260929 03:22:14.169035 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28169
I20260929 03:22:14.174788 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28303
I20260929 03:22:14.179702 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 27941
2026-09-29T03:22:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1926 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[28446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[28446](info): set up 2 sockets
Sep 29 03:22:14 dist-test-slave-fzwk krb5kdc[28446](info): commencing operation
krb5kdc: starting...
W20260929 03:22:15.211115 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:15.330605 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43025
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:15.413467 28459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:15.413686 28459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:15.413708 28459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:15.414947 28459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:15.415037 28459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:15.415068 28459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:15.415102 28459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:15.415123 28459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:15.416580 28459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:43025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:15.416807 28459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:15.416985 28459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:15.421902 28464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.421890 28467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.421877 28465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:15.422133 28459 server_base.cc:1034] running on GCE node
I20260929 03:22:15.422430 28459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:15.422757 28459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:15.423925 28459 hybrid_clock.cc:648] HybridClock initialized: now 1790652135423916 us; error 34 us; skew 500 ppm
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:15.426488 28459 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:15.426952 28459 webserver.cc:460] Webserver started at http://127.27.68.126:38043/ using document root <none> and password file <none>
I20260929 03:22:15.427203 28459 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:15.427322 28459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:15.427479 28459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:15.428668 28459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0dbc595056724e19a057ba4defe58025"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "acc674e1ef65c7d6fc60d24627cf427d"
server_key_iv: "7859dfc42a9ee2e719874bf1cd8635fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.429044 28459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0dbc595056724e19a057ba4defe58025"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "acc674e1ef65c7d6fc60d24627cf427d"
server_key_iv: "7859dfc42a9ee2e719874bf1cd8635fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.430423 28459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:15.431188 28474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.431378 28459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:15.431479 28459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0dbc595056724e19a057ba4defe58025"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "acc674e1ef65c7d6fc60d24627cf427d"
server_key_iv: "7859dfc42a9ee2e719874bf1cd8635fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.431596 28459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:15.441432 28459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:15.443475 28459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:15.443645 28459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:15.447374 28459 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:43025
I20260929 03:22:15.447439 28526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:43025 every 8 connection(s)
I20260929 03:22:15.447880 28459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:15.448807 28527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:15.451169 28527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025: Bootstrap starting.
I20260929 03:22:15.451848 28527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:15.452167 28527 log.cc:824] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:15.452917 28527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025: No bootstrap required, opened a new log
I20260929 03:22:15.454543 28527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } }
I20260929 03:22:15.454690 28527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:15.454751 28527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc595056724e19a057ba4defe58025, State: Initialized, Role: FOLLOWER
I20260929 03:22:15.454936 28527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } }
I20260929 03:22:15.455010 28527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:15.455035 28527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:15.455112 28527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:15.455647 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28459
I20260929 03:22:15.455719 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:15.455813 28527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } }
I20260929 03:22:15.455880 27921 external_mini_cluster.cc:1428] Setting key 86ec5ecbc54fedfcd64af86c0de56857
I20260929 03:22:15.455935 28527 leader_election.cc:302] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dbc595056724e19a057ba4defe58025; no voters:
I20260929 03:22:15.456151 28527 leader_election.cc:288] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:15.456173 28530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:15.456300 28530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc595056724e19a057ba4defe58025, State: Running, Role: LEADER
I20260929 03:22:15.456386 28530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } }
I20260929 03:22:15.456442 28527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:15.456745 28530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dbc595056724e19a057ba4defe58025. Latest consensus state: current_term: 1 leader_uuid: "0dbc595056724e19a057ba4defe58025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } } }
I20260929 03:22:15.456826 28530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:15.456903 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dbc595056724e19a057ba4defe58025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dbc595056724e19a057ba4defe58025" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43025 } } }
I20260929 03:22:15.456996 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:15.457069 28537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:15.457653 28537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:15.459424 28537 catalog_manager.cc:1347] Generated new cluster ID: e2c7a92bc74146bf8bc79af6682b832b
I20260929 03:22:15.459482 28537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:15.467676 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.456392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.27.68.126:43025)
0929 03:22:15.457297 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:15.457307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:15.457339 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:22:15.457476 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:15.457484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:15.457546 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:15.457705 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:15.458621 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.459890 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.460715 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.461173 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.461315 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.463947 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:15.463952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:15.463953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:15.463974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:15.465847 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.466283 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.466287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.466287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.466413 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.466658 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.466659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.466660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.466946 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:15.467163 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:15.467262 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:22:15.467423 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":63,"threads_started":1}
I20260929 03:22:15.480829 28537 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:15.481552 28537 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:15.491889 28537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dbc595056724e19a057ba4defe58025: Generated new TSK 0
I20260929 03:22:15.492136 28537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:15.521934 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:15.618892 28553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:15.619097 28553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:15.619130 28553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:15.620396 28553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:15.620446 28553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:15.620476 28553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:15.622031 28553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:15.622309 28553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:15.622555 28553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:22:15.627684 28553 server_base.cc:1034] running on GCE node
W20260929 03:22:15.627664 28558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.627672 28559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.627701 28561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:15.628019 28553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:15.628293 28553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:15.629447 28553 hybrid_clock.cc:648] HybridClock initialized: now 1790652135629434 us; error 30 us; skew 500 ppm
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:15.632081 28553 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:15.632550 28553 webserver.cc:460] Webserver started at http://127.27.68.65:46729/ using document root <none> and password file <none>
I20260929 03:22:15.632797 28553 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:15.632866 28553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:15.633018 28553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:15.634125 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "713d36078f8c4143b3fc426b7908eee1"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "7e7e9b74f8ea23d2e96cc9b0edf95803"
server_key_iv: "787275a19695fa2627164dbf4ade46e6"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.634552 28553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "713d36078f8c4143b3fc426b7908eee1"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "7e7e9b74f8ea23d2e96cc9b0edf95803"
server_key_iv: "787275a19695fa2627164dbf4ade46e6"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.636072 28553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 03:22:15.636842 28568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.637018 28553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:15.637135 28553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "713d36078f8c4143b3fc426b7908eee1"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "7e7e9b74f8ea23d2e96cc9b0edf95803"
server_key_iv: "787275a19695fa2627164dbf4ade46e6"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.637216 28553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:15.650738 28553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:15.652839 28553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:15.653023 28553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:15.653276 28553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:15.653600 28553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:15.653656 28553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.653690 28553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:15.653717 28553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.659600 28553 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:44141
I20260929 03:22:15.659646 28681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:44141 every 8 connection(s)
I20260929 03:22:15.660003 28553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:15.667743 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28553
I20260929 03:22:15.667852 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:15.668022 27921 external_mini_cluster.cc:1428] Setting key 5454b15ed2c009f8c346e39ac7d37229
I20260929 03:22:15.669859 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.660743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43025 (local address 127.27.68.65:42493)
0929 03:22:15.660931 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:15.660970 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:15.661898 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:15.662166 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:15.662170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:15.662434 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:15.662649 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.662654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.663363 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.663363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.664217 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.664220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.664583 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.664584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.664698 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.665219 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:15.665227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:15.666629 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:15.668387 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:15.668396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:15.668398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:15.668729 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:15.668950 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:15.668951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:15.668951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:15.669062 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:15.669364 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:15.669368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:15.669485 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 03:22:15.669571 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:22:15.669694 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260929 03:22:15.669852 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.660791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:42493 (local address 127.27.68.126:43025)
0929 03:22:15.660878 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:15.660881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:15.661058 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:22:15.662003 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:15.662005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:15.662025 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:15.662082 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:15.662732 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.663275 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.664304 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.664496 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.664548 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.666724 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:15.666726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:15.666727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:15.666733 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:15.668282 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.668813 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.668814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.668815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.668881 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.669129 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.669130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.669131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.669287 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:15.669371 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:15.669616 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:22:15.669712 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:22:15.670622 28682 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43025
I20260929 03:22:15.670743 28682 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:15.670924 28682 heartbeater.cc:511] Master 127.27.68.126:43025 requested a full tablet report, sending...
I20260929 03:22:15.671502 28491 ts_manager.cc:194] Registered new tserver with Master: 713d36078f8c4143b3fc426b7908eee1 (127.27.68.65:44141)
I20260929 03:22:15.672140 28491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:42493
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:15.693271 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:15.777490 28687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:15.777787 28687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:15.777833 28687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:15.779078 28687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:15.779170 28687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:15.779218 28687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:15.780948 28687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:15.781276 28687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:15.781549 28687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:15.787106 28693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.787091 28695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:15.787127 28687 server_base.cc:1034] running on GCE node
W20260929 03:22:15.787088 28692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:15.787539 28687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:15.787878 28687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:15.789040 28687 hybrid_clock.cc:648] HybridClock initialized: now 1790652135789022 us; error 38 us; skew 500 ppm
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:15.791741 28687 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:15.792168 28687 webserver.cc:460] Webserver started at http://127.27.68.66:34583/ using document root <none> and password file <none>
I20260929 03:22:15.792398 28687 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:15.792443 28687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:15.792537 28687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:15.793666 28687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "78abc4a7f81f425faa76a62daa595841"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "631d4cc8115f219303bab2935cc8d185"
server_key_iv: "cdfc64010f3805db6b6327822a98fcfa"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.794039 28687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "78abc4a7f81f425faa76a62daa595841"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "631d4cc8115f219303bab2935cc8d185"
server_key_iv: "cdfc64010f3805db6b6327822a98fcfa"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.795426 28687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:22:15.796128 28702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.796303 28687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.796392 28687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "78abc4a7f81f425faa76a62daa595841"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "631d4cc8115f219303bab2935cc8d185"
server_key_iv: "cdfc64010f3805db6b6327822a98fcfa"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.796453 28687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:15.827566 28687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:15.829628 28687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:15.829749 28687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:15.829945 28687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:15.830276 28687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:15.830310 28687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.830358 28687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:15.830386 28687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.836510 28687 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:38627
I20260929 03:22:15.836582 28815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:38627 every 8 connection(s)
I20260929 03:22:15.836923 28687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:15.838466 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28687
I20260929 03:22:15.838564 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:15.838721 27921 external_mini_cluster.cc:1428] Setting key 493766e23b750bb9299098b976e2fbaf
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
I20260929 03:22:15.847440 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.837712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:57867 (local address 127.27.68.126:43025)
0929 03:22:15.837761 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:15.837763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:15.838008 (+ 245us) server_negotiation.cc:407] Connection header received
0929 03:22:15.839102 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:15.839103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:15.839116 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:15.839170 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:15.839852 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.840414 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.841569 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.841769 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.841875 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.844440 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:15.844442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:15.844442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:15.844453 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:15.845886 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.846335 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.846337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.846338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.846402 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:15.846715 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:15.846717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:15.846717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:15.846847 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:15.846929 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:15.847245 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 03:22:15.847341 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:15.847664 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.837689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43025 (local address 127.27.68.66:57867)
0929 03:22:15.837878 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:15.837917 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:15.838977 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:15.839267 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:15.839271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:15.839546 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:15.839744 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.839749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.840519 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.840520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.841481 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.841483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.841883 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.841884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.842020 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.842625 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:15.842637 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:15.844330 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:15.845998 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:15.846007 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:15.846010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:15.846254 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:15.846472 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:15.846473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:15.846474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:15.846623 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:15.846918 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:15.846924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:15.847081 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 03:22:15.847197 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:22:15.847402 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260929 03:22:15.848206 28816 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43025
I20260929 03:22:15.848361 28816 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:15.848623 28816 heartbeater.cc:511] Master 127.27.68.126:43025 requested a full tablet report, sending...
I20260929 03:22:15.849020 28491 ts_manager.cc:194] Registered new tserver with Master: 78abc4a7f81f425faa76a62daa595841 (127.27.68.66:38627)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
I20260929 03:22:15.849534 28491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:57867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:15.863595 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:15.945281 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:15.945515 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:15.945545 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:15.946761 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:15.946810 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:15.946858 28821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:15.948479 28821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43025
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:15.948743 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:15.948994 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:15.954025 28827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.954042 28826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:15.954034 28829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:15.954278 28821 server_base.cc:1034] running on GCE node
I20260929 03:22:15.954512 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:15.954831 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:15.956014 28821 hybrid_clock.cc:648] HybridClock initialized: now 1790652135955995 us; error 30 us; skew 500 ppm
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:15.958772 28821 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:15.959332 28821 webserver.cc:460] Webserver started at http://127.27.68.67:42183/ using document root <none> and password file <none>
I20260929 03:22:15.959592 28821 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:15.959671 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:15.959831 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:15.960964 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "f4f08ab340974059a763df5aa56fe749"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "81e70394ed294aaa3616419488018e18"
server_key_iv: "46859f5c546a284a1c9858f31ee02887"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.961366 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "f4f08ab340974059a763df5aa56fe749"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "81e70394ed294aaa3616419488018e18"
server_key_iv: "46859f5c546a284a1c9858f31ee02887"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.962888 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:15.963732 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.963902 28821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.964015 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "f4f08ab340974059a763df5aa56fe749"
format_stamp: "Formatted at 2026-09-29 03:22:15 on dist-test-slave-fzwk"
server_key: "81e70394ed294aaa3616419488018e18"
server_key_iv: "46859f5c546a284a1c9858f31ee02887"
server_key_version: "encryptionkey@0"
I20260929 03:22:15.964097 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:15.983333 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:15.985428 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:15.985620 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:15.985888 28821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:15.986250 28821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:15.986310 28821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.986369 28821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:15.986418 28821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:15.993279 28821 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:36737
I20260929 03:22:15.993355 28949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:36737 every 8 connection(s)
I20260929 03:22:15.993721 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:15.998492 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 28821
I20260929 03:22:15.998620 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:15.998766 27921 external_mini_cluster.cc:1428] Setting key abcd29bec70360801c3c6bbea22ba432
Sep 29 03:22:15 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:16.003607 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.994526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:33753 (local address 127.27.68.126:43025)
0929 03:22:15.994613 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:15.994616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:15.994812 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:22:15.995837 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:15.995839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:15.995851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:15.995923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:15.996587 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.997115 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.998109 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.998356 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.998432 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.001057 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:16.001059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:16.001059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:16.001067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:16.002311 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.002708 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.002710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.002710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.002762 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.002978 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.002978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.002978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.003086 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:16.003146 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.003458 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.003533 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:22:16.003834 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:15.994485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43025 (local address 127.27.68.67:33753)
0929 03:22:15.994686 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:15.994726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:15.995728 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:15.996011 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:15.996014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:15.996278 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:15.996478 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.996482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.997202 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.997203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.998025 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:15.998027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:15.998443 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:15.998443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:15.998543 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:15.999313 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:15.999325 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:16.000967 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:16.002387 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:16.002393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:16.002395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:16.002640 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:16.002818 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:16.002819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:16.002819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:16.002924 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:16.003138 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:16.003142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:16.003333 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.003428 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.003564 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":108,"threads_started":1}
I20260929 03:22:16.004375 28950 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43025
I20260929 03:22:16.004479 28950 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:16.004645 28950 heartbeater.cc:511] Master 127.27.68.126:43025 requested a full tablet report, sending...
I20260929 03:22:16.005015 28491 ts_manager.cc:194] Registered new tserver with Master: f4f08ab340974059a763df5aa56fe749 (127.27.68.67:36737)
I20260929 03:22:16.005409 28491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:33753
I20260929 03:22:16.010740 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:22:16.019011 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.012508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.27.68.126:43025)
0929 03:22:16.012588 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.012591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.012688 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:22:16.012781 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.012782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.012793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.012845 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:16.013426 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.013923 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.014684 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.014878 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.014964 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.016274 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:16.016276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:16.016277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:16.016285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:16.017632 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.018005 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.018006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.018006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.018073 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.018290 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.018291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.018292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.018463 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:16.018581 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.018812 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.018902 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:22:16.021749 28491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:16.023401 28491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:16.030054 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.025810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:38627 (local address 127.0.0.1:43400)
0929 03:22:16.026073 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:16.026084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:16.026185 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:16.026411 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:16.026415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:16.026418 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:16.026499 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.026501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.027610 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.027612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.028883 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.028886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.029683 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.029769 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.029784 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.029886 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.029946 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":60,"threads_started":1}
I20260929 03:22:16.030040 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.025905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.27.68.66:38627)
0929 03:22:16.026059 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.026064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.026142 (+ 78us) server_negotiation.cc:407] Connection header received
0929 03:22:16.026283 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.026286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.026306 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.026369 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:16.026587 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.027508 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.028980 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.029592 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.029742 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.029904 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.029946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260929 03:22:16.030318 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.026160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.27.68.67:36737)
0929 03:22:16.026464 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.026468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.026479 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:16.026514 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.026516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.026532 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.026623 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:16.026847 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.027990 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029241 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.029914 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029971 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.030017 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.030201 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.030236 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":80,"threads_started":1}
I20260929 03:22:16.030332 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.025809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:36737 (local address 127.0.0.1:39656)
0929 03:22:16.025962 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:16.026015 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:16.026088 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:16.026724 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:16.026725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:16.026727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:16.026770 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.026772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.028097 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.028098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.029157 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.029159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.030044 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.030045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.030095 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.030098 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.030144 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.030238 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260929 03:22:16.030413 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.026467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.27.68.65:44141)
0929 03:22:16.026660 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.026664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.026673 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:16.026710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.026713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.026727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.026800 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:16.026999 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.028076 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029136 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.029856 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.029932 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.029986 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.030280 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.030323 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
I20260929 03:22:16.030735 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.026018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44141 (local address 127.0.0.1:40158)
0929 03:22:16.026123 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:16.026133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:16.026173 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:16.026886 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:16.026887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:16.026889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:16.026926 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.026927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.028159 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.028161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.029055 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.029057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.030023 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.030024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.030089 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.030093 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.030156 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.030218 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260929 03:22:16.031152 28750 tablet_service.cc:1467] Processing CreateTablet for tablet b82b1d6857c945c5acf48c169989fcc4 (DEFAULT_TABLE table=test-table [id=edf9bb8a79584d479a6bb178f884d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:16.031141 28884 tablet_service.cc:1467] Processing CreateTablet for tablet b82b1d6857c945c5acf48c169989fcc4 (DEFAULT_TABLE table=test-table [id=edf9bb8a79584d479a6bb178f884d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:16.031558 28750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82b1d6857c945c5acf48c169989fcc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:16.032838 28616 tablet_service.cc:1467] Processing CreateTablet for tablet b82b1d6857c945c5acf48c169989fcc4 (DEFAULT_TABLE table=test-table [id=edf9bb8a79584d479a6bb178f884d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:16.033327 28616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82b1d6857c945c5acf48c169989fcc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:16.035156 28967 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Bootstrap starting.
I20260929 03:22:16.035876 28967 tablet_bootstrap.cc:654] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:16.036367 28967 log.cc:824] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:16.036561 28968 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: Bootstrap starting.
I20260929 03:22:16.036741 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82b1d6857c945c5acf48c169989fcc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:16.037209 28968 tablet_bootstrap.cc:654] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:16.037761 28968 log.cc:824] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:16.038596 28968 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: No bootstrap required, opened a new log
I20260929 03:22:16.038672 28968 ts_tablet_manager.cc:1397] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:16.039860 28967 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: No bootstrap required, opened a new log
I20260929 03:22:16.039937 28967 ts_tablet_manager.cc:1397] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:22:16.040277 28971 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: Bootstrap starting.
I20260929 03:22:16.040792 28968 raft_consensus.cc:348] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.040958 28968 raft_consensus.cc:374] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:16.041000 28968 raft_consensus.cc:729] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713d36078f8c4143b3fc426b7908eee1, State: Initialized, Role: FOLLOWER
I20260929 03:22:16.040997 28971 tablet_bootstrap.cc:654] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:16.041108 28968 consensus_queue.cc:260] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.041404 28682 heartbeater.cc:503] Master 127.27.68.126:43025 was elected leader, sending a full tablet report...
I20260929 03:22:16.041424 28968 ts_tablet_manager.cc:1428] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:16.041523 28971 log.cc:824] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:16.042284 28971 tablet_bootstrap.cc:492] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: No bootstrap required, opened a new log
I20260929 03:22:16.042347 28971 ts_tablet_manager.cc:1397] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:16.042228 28967 raft_consensus.cc:348] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.042411 28967 raft_consensus.cc:374] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:16.042454 28967 raft_consensus.cc:729] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abc4a7f81f425faa76a62daa595841, State: Initialized, Role: FOLLOWER
I20260929 03:22:16.042584 28967 consensus_queue.cc:260] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.042867 28967 ts_tablet_manager.cc:1428] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:16.042879 28816 heartbeater.cc:503] Master 127.27.68.126:43025 was elected leader, sending a full tablet report...
I20260929 03:22:16.044144 28971 raft_consensus.cc:348] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.044296 28971 raft_consensus.cc:374] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:16.044319 28971 raft_consensus.cc:729] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f08ab340974059a763df5aa56fe749, State: Initialized, Role: FOLLOWER
I20260929 03:22:16.044445 28971 consensus_queue.cc:260] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.044700 28971 ts_tablet_manager.cc:1428] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:22:16.044747 28950 heartbeater.cc:503] Master 127.27.68.126:43025 was elected leader, sending a full tablet report...
I20260929 03:22:16.056550 28972 raft_consensus.cc:482] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:16.056702 28972 raft_consensus.cc:504] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.057097 28972 leader_election.cc:288] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78abc4a7f81f425faa76a62daa595841 (127.27.68.66:38627), f4f08ab340974059a763df5aa56fe749 (127.27.68.67:36737)
I20260929 03:22:16.061219 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.057297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:43587 (local address 127.27.68.67:36737)
0929 03:22:16.057357 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.057359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.057369 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:16.057460 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.057461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.057473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.057551 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:16.057751 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.058622 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060238 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.060830 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060910 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.060919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.061102 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.061142 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:22:16.061219 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.057410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:35997 (local address 127.27.68.66:38627)
0929 03:22:16.057452 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.057454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.057517 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:22:16.057608 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.057609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.057618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.057665 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:16.057914 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.058622 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060239 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.060828 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060911 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.060919 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.061102 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.061142 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:16.061255 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.057240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:36737 (local address 127.27.68.65:43587)
0929 03:22:16.057314 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:16.057338 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:16.057380 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:16.057572 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:16.057573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:16.057575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:16.057673 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.057675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.058736 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.058737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.060143 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.060147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.060926 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.061013 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.061020 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.061081 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.061136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":81}
I20260929 03:22:16.061262 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.057258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:38627 (local address 127.27.68.65:35997)
0929 03:22:16.057439 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:16.057448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:16.057547 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:16.057676 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:16.057677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:16.057678 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:16.057746 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.057747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.058736 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.058737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.060143 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:16.060147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.060926 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.060926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:16.061013 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.061020 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:22:16.061088 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:22:16.061136 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":90,"thread_start_us":116,"threads_started":1}
I20260929 03:22:16.061617 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82b1d6857c945c5acf48c169989fcc4" candidate_uuid: "713d36078f8c4143b3fc426b7908eee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f08ab340974059a763df5aa56fe749" is_pre_election: true
I20260929 03:22:16.061708 28770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82b1d6857c945c5acf48c169989fcc4" candidate_uuid: "713d36078f8c4143b3fc426b7908eee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abc4a7f81f425faa76a62daa595841" is_pre_election: true
I20260929 03:22:16.061820 28904 raft_consensus.cc:2454] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713d36078f8c4143b3fc426b7908eee1 in term 0.
I20260929 03:22:16.061864 28770 raft_consensus.cc:2454] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713d36078f8c4143b3fc426b7908eee1 in term 0.
I20260929 03:22:16.062055 28571 leader_election.cc:302] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713d36078f8c4143b3fc426b7908eee1, f4f08ab340974059a763df5aa56fe749; no voters:
I20260929 03:22:16.062218 28972 raft_consensus.cc:2781] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:16.062282 28972 raft_consensus.cc:482] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:16.062322 28972 raft_consensus.cc:3037] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:16.063086 28972 raft_consensus.cc:504] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.063271 28972 leader_election.cc:288] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [CANDIDATE]: Term 1 election: Requested vote from peers 78abc4a7f81f425faa76a62daa595841 (127.27.68.66:38627), f4f08ab340974059a763df5aa56fe749 (127.27.68.67:36737)
I20260929 03:22:16.063397 28770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82b1d6857c945c5acf48c169989fcc4" candidate_uuid: "713d36078f8c4143b3fc426b7908eee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abc4a7f81f425faa76a62daa595841"
I20260929 03:22:16.063462 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b82b1d6857c945c5acf48c169989fcc4" candidate_uuid: "713d36078f8c4143b3fc426b7908eee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4f08ab340974059a763df5aa56fe749"
I20260929 03:22:16.063494 28770 raft_consensus.cc:3037] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:16.063530 28904 raft_consensus.cc:3037] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:16.064257 28904 raft_consensus.cc:2454] T b82b1d6857c945c5acf48c169989fcc4 P f4f08ab340974059a763df5aa56fe749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713d36078f8c4143b3fc426b7908eee1 in term 1.
I20260929 03:22:16.064266 28770 raft_consensus.cc:2454] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713d36078f8c4143b3fc426b7908eee1 in term 1.
I20260929 03:22:16.064435 28571 leader_election.cc:302] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713d36078f8c4143b3fc426b7908eee1, f4f08ab340974059a763df5aa56fe749; no voters:
I20260929 03:22:16.064527 28972 raft_consensus.cc:2781] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:16.064698 28972 raft_consensus.cc:686] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [term 1 LEADER]: Becoming Leader. State: Replica: 713d36078f8c4143b3fc426b7908eee1, State: Running, Role: LEADER
I20260929 03:22:16.064806 28972 consensus_queue.cc:237] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } }
I20260929 03:22:16.065739 28490 catalog_manager.cc:5495] T b82b1d6857c945c5acf48c169989fcc4 P 713d36078f8c4143b3fc426b7908eee1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 713d36078f8c4143b3fc426b7908eee1 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "713d36078f8c4143b3fc426b7908eee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abc4a7f81f425faa76a62daa595841" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4f08ab340974059a763df5aa56fe749" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 36737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713d36078f8c4143b3fc426b7908eee1" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44141 } health_report { overall_health: HEALTHY } } }
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:16.079619 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.072630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40160 (local address 127.27.68.65:44141)
0929 03:22:16.072705 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.072707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.072729 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:22:16.072781 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.072781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.072793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.072834 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:16.073365 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.073829 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.074512 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.074696 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.074747 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.077084 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:16.077095 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:16.077096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:16.077104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:16.078541 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.078870 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.078873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.078873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.078941 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.079102 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.079102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.079102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.079304 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:16.079420 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.079473 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.079539 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 03:22:16.087491 28817 tablet.cc:2369] T b82b1d6857c945c5acf48c169989fcc4 P 78abc4a7f81f425faa76a62daa595841: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652136, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652136, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:16.102766 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.095504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.27.68.65:44141)
0929 03:22:16.095576 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.095579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.095664 (+ 85us) server_negotiation.cc:407] Connection header received
0929 03:22:16.095763 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.095764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.095775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.095834 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:16.096411 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.096909 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.097644 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.097851 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.097936 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.100274 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:16.100276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:16.100276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:16.100284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:16.101520 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.101897 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.101899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.101899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.101961 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.102164 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.102165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.102165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.102272 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:16.102376 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.102571 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.102659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:16.125437 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:16.118479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.27.68.65:44141)
0929 03:22:16.118575 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:16.118577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:16.118588 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:16.118636 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:16.118637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:16.118646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:16.118696 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:16.119279 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.119760 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.120519 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:16.120716 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:16.120767 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:16.123267 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:16.123269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:16.123270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:16.123279 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:16.124509 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.124817 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.124818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.124818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.124868 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:16.125022 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:16.125022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:16.125022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:16.125111 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:16.125182 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:16.125288 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:22:16.125368 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 03:22:16.125955 28596 tablet_service.cc:2908] Rejecting scan request for tablet b82b1d6857c945c5acf48c169989fcc4: Uninitialized: clean time has not yet been initialized
I20260929 03:22:16.126801 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28553
I20260929 03:22:16.132009 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28687
I20260929 03:22:16.136699 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28821
I20260929 03:22:16.142006 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 28459
2026-09-29T03:22:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1963 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28991](info): set up 2 sockets
Sep 29 03:22:16 dist-test-slave-fzwk krb5kdc[28991](info): commencing operation
krb5kdc: starting...
W20260929 03:22:17.265098 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 03:22:17.387447 27921 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.122s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 03:22:17.558975 27921 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.171s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 03:22:17.733188 27921 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.174s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/test-user.keytab.
W20260929 03:22:18.549856 27921 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.817s user 0.000s sys 0.002s
Sep 29 03:22:18 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
W20260929 03:22:19.525753 27921 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.68.126: real 0.954s user 0.001s sys 0.001s
I20260929 03:22:19.526229 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46697
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:19.623945 29004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:19.624164 29004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:19.624188 29004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:19.625603 29004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:19.625659 29004 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:22:19.625674 29004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:19.625686 29004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:19.625699 29004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:19.625741 29004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:19.627305 29004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46697
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:19.627580 29004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:19.627849 29004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:19.632977 29009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:19.633179 29010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:19.632977 29012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:19.633185 29004 server_base.cc:1034] running on GCE node
I20260929 03:22:19.633675 29004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:19.634022 29004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:19.635195 29004 hybrid_clock.cc:648] HybridClock initialized: now 1790652139635169 us; error 41 us; skew 500 ppm
Sep 29 03:22:19 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:19.637844 29004 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:19.638365 29004 webserver.cc:460] Webserver started at http://127.27.68.126:43537/ using document root <none> and password file <none>
I20260929 03:22:19.638604 29004 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:19.638677 29004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:19.638839 29004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:19.640057 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "c9307d90bea649b0b1e343db7e9fae4f"
format_stamp: "Formatted at 2026-09-29 03:22:19 on dist-test-slave-fzwk"
server_key: "77e1be438965479545f4d5900962886a"
server_key_iv: "e98cadd06ebf6f63aaea013c8e7fb753"
server_key_version: "encryptionkey@0"
I20260929 03:22:19.640434 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "c9307d90bea649b0b1e343db7e9fae4f"
format_stamp: "Formatted at 2026-09-29 03:22:19 on dist-test-slave-fzwk"
server_key: "77e1be438965479545f4d5900962886a"
server_key_iv: "e98cadd06ebf6f63aaea013c8e7fb753"
server_key_version: "encryptionkey@0"
I20260929 03:22:19.641863 29004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:22:19.642674 29019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:19.642894 29004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:19.643007 29004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "c9307d90bea649b0b1e343db7e9fae4f"
format_stamp: "Formatted at 2026-09-29 03:22:19 on dist-test-slave-fzwk"
server_key: "77e1be438965479545f4d5900962886a"
server_key_iv: "e98cadd06ebf6f63aaea013c8e7fb753"
server_key_version: "encryptionkey@0"
I20260929 03:22:19.643086 29004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:19.669870 29004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:19.672139 29004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:19.672334 29004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:19.677378 29004 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46697
I20260929 03:22:19.677435 29081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46697 every 8 connection(s)
I20260929 03:22:19.677798 29004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:19.678473 29082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:19.680913 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Bootstrap starting.
I20260929 03:22:19.681593 29082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:19.681923 29082 log.cc:824] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:19.682664 29082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: No bootstrap required, opened a new log
I20260929 03:22:19.684824 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29004
I20260929 03:22:19.684930 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:19.685097 27921 external_mini_cluster.cc:1428] Setting key 5dcb9469a34f6dbf6fdeffba2348a240
I20260929 03:22:19.685118 29082 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } }
I20260929 03:22:19.685339 29082 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:19.685395 29082 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9307d90bea649b0b1e343db7e9fae4f, State: Initialized, Role: FOLLOWER
I20260929 03:22:19.685570 29082 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } }
I20260929 03:22:19.685702 29082 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:19.685761 29082 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:19.685859 29082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 0 FOLLOWER]: Advancing to term 1
Sep 29 03:22:19 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:19.694454 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:19.685634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.27.68.126:46697)
0929 03:22:19.685851 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:19.685862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:19.685899 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:22:19.686039 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:19.686048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:19.686108 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:19.686273 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:19.686841 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:19.687827 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:19.688558 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:19.688947 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:19.689028 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:19.691369 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:19.691374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:19.691374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:19.691386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:19.693044 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:19.693411 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:19.693414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:19.693414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:19.693522 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:19.693696 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:19.693697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:19.693697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:19.693952 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:19.694060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:19.694167 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:22:19.694279 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260929 03:22:19.736126 29082 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } }
I20260929 03:22:19.736384 29082 leader_election.cc:302] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9307d90bea649b0b1e343db7e9fae4f; no voters:
I20260929 03:22:19.736661 29082 leader_election.cc:288] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:19.736752 29087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:19.736961 29087 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [term 1 LEADER]: Becoming Leader. State: Replica: c9307d90bea649b0b1e343db7e9fae4f, State: Running, Role: LEADER
I20260929 03:22:19.737010 29082 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:19.737174 29087 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } }
I20260929 03:22:19.737658 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9307d90bea649b0b1e343db7e9fae4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } } }
I20260929 03:22:19.737784 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:19.738588 29089 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9307d90bea649b0b1e343db7e9fae4f. Latest consensus state: current_term: 1 leader_uuid: "c9307d90bea649b0b1e343db7e9fae4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9307d90bea649b0b1e343db7e9fae4f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46697 } } }
I20260929 03:22:19.738588 29093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:19.739344 29089 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:19.739679 29093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:19.741834 29093 catalog_manager.cc:1347] Generated new cluster ID: 4295392fb7954cc5a44fd9c6426bc028
I20260929 03:22:19.741899 29093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:19.752676 29093 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:19.753280 29093 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:19.759123 29093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Generated new TSK 0
I20260929 03:22:19.759374 29093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
W20260929 03:22:20.841078 27921 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.68.65: real 1.074s user 0.001s sys 0.002s
I20260929 03:22:20.841403 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46697
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:20.927984 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:20.928292 29108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:20.928335 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:20.929651 29108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:20.929736 29108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:20.929774 29108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:20.929840 29108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:20.931448 29108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:20.931751 29108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:20.932013 29108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:20.937402 29114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:20.937374 29113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:20.937796 29108 server_base.cc:1034] running on GCE node
W20260929 03:22:20.937374 29116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:20.938136 29108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:20.938476 29108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:20.939661 29108 hybrid_clock.cc:648] HybridClock initialized: now 1790652140939645 us; error 33 us; skew 500 ppm
Sep 29 03:22:20 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652140, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:20.942412 29108 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:20.942950 29108 webserver.cc:460] Webserver started at http://127.27.68.65:41279/ using document root <none> and password file <none>
I20260929 03:22:20.943202 29108 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:20.943336 29108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:20.943480 29108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:20.944597 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "9afd1fd787764319b97f3729bd920eaa"
format_stamp: "Formatted at 2026-09-29 03:22:20 on dist-test-slave-fzwk"
server_key: "02302c07127db2b294e1cc9b885203f0"
server_key_iv: "08a4d09d3895bccaf4add23e29d09569"
server_key_version: "encryptionkey@0"
I20260929 03:22:20.944983 29108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "9afd1fd787764319b97f3729bd920eaa"
format_stamp: "Formatted at 2026-09-29 03:22:20 on dist-test-slave-fzwk"
server_key: "02302c07127db2b294e1cc9b885203f0"
server_key_iv: "08a4d09d3895bccaf4add23e29d09569"
server_key_version: "encryptionkey@0"
I20260929 03:22:20.946586 29108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:20.947382 29123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:20.947608 29108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:22:20.947690 29108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "9afd1fd787764319b97f3729bd920eaa"
format_stamp: "Formatted at 2026-09-29 03:22:20 on dist-test-slave-fzwk"
server_key: "02302c07127db2b294e1cc9b885203f0"
server_key_iv: "08a4d09d3895bccaf4add23e29d09569"
server_key_version: "encryptionkey@0"
I20260929 03:22:20.947747 29108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:20.971158 29108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:20.973295 29108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:20.973418 29108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:20.973910 29108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:20.973980 29108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:20.974023 29108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:20.974066 29108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:20.981410 29108 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:37525
I20260929 03:22:20.981493 29239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:37525 every 8 connection(s)
Sep 29 03:22:20 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652140, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:20.981855 29108 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:20.984944 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:20.974507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:48753 (local address 127.27.68.126:46697)
0929 03:22:20.974642 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:20.974644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:20.974827 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:22:20.975457 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:20.975459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:20.975474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:20.975541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:20.976230 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.977078 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.978859 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.979204 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.979337 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:20.982309 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:20.982311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:20.982311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:20.982320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:20.983733 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:20.984123 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:20.984124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:20.984124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:20.984181 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:20.984397 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:20.984398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:20.984398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:20.984507 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:20.984558 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:20.984798 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:22:20.984870 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260929 03:22:20.985105 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:20.974516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.65:48753)
0929 03:22:20.974693 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:20.974743 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:20.975351 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:20.975614 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:20.975619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:20.975909 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:20.976117 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:20.976129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.977187 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.977188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:20.978731 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:20.978736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.979348 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.979349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:20.979498 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:20.980125 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:20.980136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:20.982177 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:20.983812 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:20.983818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:20.983820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:20.984052 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:20.984240 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:20.984240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:20.984240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:20.984343 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:20.984561 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:20.984564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:20.984688 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 03:22:20.984772 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:22:20.984900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260929 03:22:20.985647 29240 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46697
I20260929 03:22:20.985737 29240 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:20.985948 29240 heartbeater.cc:511] Master 127.27.68.126:46697 requested a full tablet report, sending...
I20260929 03:22:20.986570 29035 ts_manager.cc:194] Registered new tserver with Master: 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525)
I20260929 03:22:20.986698 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29108
I20260929 03:22:20.986788 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:20.986960 27921 external_mini_cluster.cc:1428] Setting key 281a062d38579898becbe6b1a27829da
I20260929 03:22:20.987303 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:48753
I20260929 03:22:20.994326 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:20.987719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34093 (local address 127.27.68.126:46697)
0929 03:22:20.987760 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:20.987762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:20.988352 (+ 590us) server_negotiation.cc:407] Connection header received
0929 03:22:20.988427 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:20.988428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:20.988439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:20.988487 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:20.988881 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.989351 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.990387 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.990568 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.990617 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:20.991757 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:20.991759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:20.991759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:20.991765 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:20.993205 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:20.993622 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:20.993623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:20.993623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:20.993673 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:20.993874 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:20.993875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:20.993875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:20.993973 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:20.994022 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:20.994193 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:22:20.994259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:22:20.994421 29246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:20.987670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.65:34093)
0929 03:22:20.988281 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:20.988293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:20.988369 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:20.988494 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:20.988495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:20.988735 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:20.988806 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:20.988808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.989455 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.989456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:20.990304 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:20.990307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:20.990644 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:20.990645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:20.990735 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:20.991180 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:20.991185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:20.991676 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:20.993290 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:20.993291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:20.993292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:20.993532 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:20.993738 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:20.993739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:20.993739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:20.993810 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:20.994036 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:20.994038 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:20.994094 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 03:22:20.994180 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:22:20.994298 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":519,"threads_started":1}
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
W20260929 03:22:21.149073 27921 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.68.66: real 0.161s user 0.002s sys 0.001s
I20260929 03:22:21.149435 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46697
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:21.229933 29249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:21.230137 29249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:21.230161 29249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:21.231392 29249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:21.231431 29249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:21.231453 29249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:21.231513 29249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:21.232973 29249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:21.233227 29249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:21.233461 29249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:21.238528 29254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:21.238546 29257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:21.238548 29255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:21.238749 29249 server_base.cc:1034] running on GCE node
I20260929 03:22:21.238968 29249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:21.239282 29249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:21.240454 29249 hybrid_clock.cc:648] HybridClock initialized: now 1790652141240443 us; error 51 us; skew 500 ppm
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:21.242942 29249 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:21.243484 29249 webserver.cc:460] Webserver started at http://127.27.68.66:43549/ using document root <none> and password file <none>
I20260929 03:22:21.243733 29249 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:21.243806 29249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:21.243952 29249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:21.245059 29249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "64dfb0c040964dab86428a4fd093d2f7"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "5f24a583de399c72acad0c98101e27f3"
server_key_iv: "689ee814f7ec8d2f50d5fb89e1f25fdc"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.245424 29249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "64dfb0c040964dab86428a4fd093d2f7"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "5f24a583de399c72acad0c98101e27f3"
server_key_iv: "689ee814f7ec8d2f50d5fb89e1f25fdc"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.246855 29249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:22:21.247741 29264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.247910 29249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.248010 29249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "64dfb0c040964dab86428a4fd093d2f7"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "5f24a583de399c72acad0c98101e27f3"
server_key_iv: "689ee814f7ec8d2f50d5fb89e1f25fdc"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.248086 29249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:21.259590 29249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:21.261482 29249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:21.261644 29249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:21.262171 29249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:21.262219 29249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.262243 29249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:21.262256 29249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.269448 29249 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:39419
I20260929 03:22:21.269492 29379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:39419 every 8 connection(s)
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:21.269865 29249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:21.273047 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.262688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:39769 (local address 127.27.68.126:46697)
0929 03:22:21.262742 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.262744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.262921 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:22:21.263591 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.263592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.263604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.263656 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:21.264601 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.265466 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.267085 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.267436 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.267488 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.270312 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:21.270315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:21.270316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:21.270324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:21.271862 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.272262 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.272262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.272263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.272314 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.272524 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.272525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.272525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.272629 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:21.272678 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.272912 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.272981 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:22:21.273212 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.262644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.66:39769)
0929 03:22:21.262806 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.262848 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.263486 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.263807 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.263812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.264190 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:21.264482 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.264489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.265600 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.265602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.266965 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.266969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.267584 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.267585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.267737 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.268316 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:21.268326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:21.270189 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:21.271938 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.271944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.271945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.272178 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.272371 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.272371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.272371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.272469 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.272682 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:21.272686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:21.272800 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.272885 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.273017 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260929 03:22:21.273780 29380 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46697
I20260929 03:22:21.273867 29380 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:21.274057 29380 heartbeater.cc:511] Master 127.27.68.126:46697 requested a full tablet report, sending...
I20260929 03:22:21.274355 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29249
I20260929 03:22:21.274464 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:21.274487 29035 ts_manager.cc:194] Registered new tserver with Master: 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419)
I20260929 03:22:21.274623 27921 external_mini_cluster.cc:1428] Setting key 750e8fa9f413b658868726b23a340dd9
I20260929 03:22:21.274942 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:39769
I20260929 03:22:21.281426 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.274996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.66:56351)
0929 03:22:21.275353 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.275364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.275431 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.275567 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.275568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.275785 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:21.275855 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.275857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.276493 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.276494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.277131 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.277133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.277603 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.277604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.277656 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.278109 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:21.278113 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:21.278400 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:21.280291 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.280292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.280293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.280465 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.280683 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.280683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.280683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.280737 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.281140 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:21.281141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:21.281187 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.281256 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.281350 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":159,"threads_started":1}
I20260929 03:22:21.281500 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.275164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:56351 (local address 127.27.68.126:46697)
0929 03:22:21.275200 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.275233 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.275409 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:22:21.275480 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.275481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.275491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.275559 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:21.275940 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.276395 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.277212 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.277527 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.277602 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.278496 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:21.278498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:21.278499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:21.278506 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:21.280194 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.280543 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.280544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.280545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.280614 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.280800 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.280802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.280802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.281043 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:21.281127 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.281307 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.281396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:21.301751 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46697
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:21.386314 29389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:21.386531 29389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:21.386556 29389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:21.387743 29389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:21.387780 29389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:21.387801 29389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:21.387818 29389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:21.389223 29389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:21.389427 29389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:21.389611 29389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:21.394673 29397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:21.394769 29389 server_base.cc:1034] running on GCE node
W20260929 03:22:21.394702 29395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:21.394680 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:21.395099 29389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:21.395402 29389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:21.396565 29389 hybrid_clock.cc:648] HybridClock initialized: now 1790652141396566 us; error 47 us; skew 500 ppm
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:21.399032 29389 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:21.399572 29389 webserver.cc:460] Webserver started at http://127.27.68.67:36179/ using document root <none> and password file <none>
I20260929 03:22:21.399830 29389 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:21.399899 29389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:21.400050 29389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:21.401118 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "c4205ebd0af04d568f47a5770eabeef0"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "0a37a05271076e00ad6e0ed4700a279f"
server_key_iv: "36e6ce49bb1c330c2f1c7a7b4aef50ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.401486 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "c4205ebd0af04d568f47a5770eabeef0"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "0a37a05271076e00ad6e0ed4700a279f"
server_key_iv: "36e6ce49bb1c330c2f1c7a7b4aef50ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.402817 29389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:21.403594 29404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.403821 29389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:22:21.403928 29389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "c4205ebd0af04d568f47a5770eabeef0"
format_stamp: "Formatted at 2026-09-29 03:22:21 on dist-test-slave-fzwk"
server_key: "0a37a05271076e00ad6e0ed4700a279f"
server_key_iv: "36e6ce49bb1c330c2f1c7a7b4aef50ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:21.404002 29389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:21.419523 29389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:21.421574 29389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:21.421764 29389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:21.422361 29389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:21.422430 29389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.422465 29389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:21.422494 29389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:21.429831 29389 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:34789
I20260929 03:22:21.429968 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:34789 every 8 connection(s)
I20260929 03:22:21.430267 29389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:21.432680 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.422756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:36725 (local address 127.27.68.126:46697)
0929 03:22:21.422827 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.422830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.423069 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:22:21.423648 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.423650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.423662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.423718 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:21.424387 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.424943 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.425955 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.426147 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.426257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.429636 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:21.429639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:21.429639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:21.429650 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:21.431465 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.431854 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.431854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.431855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.431911 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.432123 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.432123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.432124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.432235 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:21.432298 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.432546 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.432614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:22:21.432863 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.422764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.67:36725)
0929 03:22:21.422925 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.422967 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.423544 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.423798 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.423802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.424060 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:21.424297 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.424302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.425047 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.425048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.425858 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.425859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.426268 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.426270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.426417 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.427028 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:21.427038 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:21.429523 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:21.431555 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.431560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.431562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.431788 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.431969 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.431970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.431970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.432068 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.432301 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:21.432305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:21.432427 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.432516 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.432648 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260929 03:22:21.433558 29520 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46697
I20260929 03:22:21.433640 29520 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:21.433845 29520 heartbeater.cc:511] Master 127.27.68.126:46697 requested a full tablet report, sending...
I20260929 03:22:21.434296 29035 ts_manager.cc:194] Registered new tserver with Master: c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67:34789)
I20260929 03:22:21.434762 29035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:36725
I20260929 03:22:21.436955 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29389
I20260929 03:22:21.437065 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:21.437260 27921 external_mini_cluster.cc:1428] Setting key 201d8a785b2d442a874424fe5a200db5
I20260929 03:22:21.438139 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:22:21.441151 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.434751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:47877 (local address 127.27.68.126:46697)
0929 03:22:21.434814 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.434817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.434914 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:22:21.434994 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.434994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.435007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.435055 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:21.435436 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.435995 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.436715 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.437002 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.437083 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.438419 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:21.438420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:21.438420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:21.438428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:21.439943 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.440321 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.440322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.440322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.440386 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.440609 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.440610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.440610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.440734 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:21.440805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.440967 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.441051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:22:21.441174 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.434664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.27.68.67:47877)
0929 03:22:21.434852 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.434864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.434939 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.435069 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.435069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.435293 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:21.435362 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.435364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.436086 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.436086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.436593 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.436595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.437117 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.437118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.437203 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.437867 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:21.437872 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:21.438304 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:21.440049 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.440051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.440051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.440237 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.440460 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:21.440461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:21.440461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:21.440541 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:21.440807 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:21.440808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:21.440858 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.440942 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.441053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":87,"threads_started":1}
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:22:21 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:21.461527 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.454646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.27.68.126:46697)
0929 03:22:21.454728 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.454730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.454744 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:22:21.454843 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.454844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.454855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.454928 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:21.455584 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.456072 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.456801 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.456991 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.457073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.459367 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:21.459369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:21.459369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:21.459376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:21.460593 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.460910 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.460911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.460911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.460960 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:21.461117 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:21.461118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:21.461118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:21.461222 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:21.461299 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.461387 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.461446 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:22:21.463142 29035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:21.464764 29035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:21.471751 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.27.68.66:39419)
0929 03:22:21.467495 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.467500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.467662 (+ 162us) server_negotiation.cc:407] Connection header received
0929 03:22:21.467752 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.467755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.467773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.467853 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.468039 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469201 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.470467 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471309 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.471392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.471461 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.471604 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.471641 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260929 03:22:21.471740 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37525 (local address 127.0.0.1:47016)
0929 03:22:21.467775 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.467784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.467823 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.468190 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.468191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.468193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.468251 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.468253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469117 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.469119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.470482 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.470484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471450 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.471451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.471519 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.471523 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.471581 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.471631 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260929 03:22:21.471742 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39419 (local address 127.0.0.1:33136)
0929 03:22:21.467540 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.467581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.467662 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.467857 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.467859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.467862 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.467937 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.467940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469285 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.469286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.470372 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.470374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471400 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.471401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.471480 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.471494 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.471566 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.471637 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":54,"thread_start_us":123,"threads_started":1}
I20260929 03:22:21.471832 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.27.68.67:34789)
0929 03:22:21.467435 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.467440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.467667 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:22:21.467790 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.467793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.467808 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.467862 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.468056 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469201 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.470770 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471347 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.471431 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.471488 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.471670 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.471725 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260929 03:22:21.471994 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34789 (local address 127.0.0.1:42806)
0929 03:22:21.467545 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.467602 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.467698 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.467945 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.467945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.467947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.467980 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.467981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469677 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.469677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.470690 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.470691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471520 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.471521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.471582 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.471585 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.471896 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.471935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":125,"threads_started":1}
I20260929 03:22:21.472723 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.467665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.27.68.65:37525)
0929 03:22:21.468018 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.468023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.468032 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:21.468075 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.468077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.468093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.468324 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.468370 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.469022 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.470566 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.471359 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.472481 (+ 1122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.472554 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.472580 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.472625 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":66,"threads_started":1}
I20260929 03:22:21.472754 29454 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9f55ec8e8d47f3959084440c64754b (DEFAULT_TABLE table=test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.472980 29314 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9f55ec8e8d47f3959084440c64754b (DEFAULT_TABLE table=test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.473152 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9f55ec8e8d47f3959084440c64754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.473318 29314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9f55ec8e8d47f3959084440c64754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.475665 29174 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9f55ec8e8d47f3959084440c64754b (DEFAULT_TABLE table=test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.476011 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9f55ec8e8d47f3959084440c64754b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.476179 29544 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Bootstrap starting.
I20260929 03:22:21.476894 29545 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Bootstrap starting.
I20260929 03:22:21.477114 29544 tablet_bootstrap.cc:654] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.477361 29544 log.cc:824] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:21.477537 29545 tablet_bootstrap.cc:654] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.477856 29545 log.cc:824] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:21.478039 29544 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: No bootstrap required, opened a new log
I20260929 03:22:21.478122 29544 ts_tablet_manager.cc:1397] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:21.478641 29545 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: No bootstrap required, opened a new log
I20260929 03:22:21.478698 29545 ts_tablet_manager.cc:1397] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:21.480029 29544 raft_consensus.cc:348] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.480163 29544 raft_consensus.cc:374] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.480185 29544 raft_consensus.cc:729] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4205ebd0af04d568f47a5770eabeef0, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.480306 29544 consensus_queue.cc:260] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.480263 29545 raft_consensus.cc:348] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.480446 29545 raft_consensus.cc:374] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.480494 29545 raft_consensus.cc:729] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dfb0c040964dab86428a4fd093d2f7, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.480564 29544 ts_tablet_manager.cc:1428] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:22:21.480615 29520 heartbeater.cc:503] Master 127.27.68.126:46697 was elected leader, sending a full tablet report...
I20260929 03:22:21.480636 29545 consensus_queue.cc:260] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.480775 29548 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Bootstrap starting.
I20260929 03:22:21.480950 29380 heartbeater.cc:503] Master 127.27.68.126:46697 was elected leader, sending a full tablet report...
I20260929 03:22:21.480894 29545 ts_tablet_manager.cc:1428] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:21.481819 29548 tablet_bootstrap.cc:654] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.482139 29548 log.cc:824] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:21.482824 29548 tablet_bootstrap.cc:492] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: No bootstrap required, opened a new log
I20260929 03:22:21.482908 29548 ts_tablet_manager.cc:1397] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:21.484454 29548 raft_consensus.cc:348] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.484596 29548 raft_consensus.cc:374] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.484618 29548 raft_consensus.cc:729] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afd1fd787764319b97f3729bd920eaa, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.484735 29548 consensus_queue.cc:260] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.484958 29548 ts_tablet_manager.cc:1428] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:21.485025 29240 heartbeater.cc:503] Master 127.27.68.126:46697 was elected leader, sending a full tablet report...
W20260929 03:22:21.520484 29381 tablet.cc:2369] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:21.551016 29549 raft_consensus.cc:482] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:21.551154 29549 raft_consensus.cc:504] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.551586 29549 leader_election.cc:288] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419), 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525)
I20260929 03:22:21.555693 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.552016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:34059 (local address 127.27.68.65:37525)
0929 03:22:21.552072 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.552074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.552182 (+ 108us) server_negotiation.cc:407] Connection header received
0929 03:22:21.552272 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.552273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.552281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.552342 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.552454 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.553161 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.554718 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.555302 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.555382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.555391 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.555577 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.555618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:22:21.555693 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.551881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:52663 (local address 127.27.68.66:39419)
0929 03:22:21.551924 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.551927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.551998 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:22:21.552103 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.552104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.552113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.552189 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.552363 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.553160 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.554717 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.555297 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.555382 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.555391 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.555579 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.555617 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:21.555708 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.551843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39419 (local address 127.27.68.67:52663)
0929 03:22:21.551913 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.551925 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.552017 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.552206 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.552207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.552209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.552289 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.552291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.553259 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.553260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.554616 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.554619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.555390 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.555391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.555499 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.555502 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.555554 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.555603 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":81}
I20260929 03:22:21.555706 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.551961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37525 (local address 127.27.68.67:34059)
0929 03:22:21.552104 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.552114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.552197 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.552328 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.552328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.552330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.552391 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.552392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.553259 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.553260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.554616 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.554619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.555390 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.555391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.555487 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.555491 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.555554 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.555603 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":84,"thread_start_us":96,"threads_started":1}
I20260929 03:22:21.556077 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9f55ec8e8d47f3959084440c64754b" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7" is_pre_election: true
I20260929 03:22:21.556219 29322 raft_consensus.cc:2454] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 0.
I20260929 03:22:21.556416 29408 leader_election.cc:302] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dfb0c040964dab86428a4fd093d2f7, c4205ebd0af04d568f47a5770eabeef0; no voters:
I20260929 03:22:21.556571 29549 raft_consensus.cc:2781] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:21.556630 29549 raft_consensus.cc:482] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:21.556545 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9f55ec8e8d47f3959084440c64754b" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afd1fd787764319b97f3729bd920eaa" is_pre_election: true
I20260929 03:22:21.556666 29549 raft_consensus.cc:3037] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.556723 29194 raft_consensus.cc:2454] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 0.
I20260929 03:22:21.557443 29549 raft_consensus.cc:504] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.557592 29549 leader_election.cc:288] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 election: Requested vote from peers 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419), 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525)
I20260929 03:22:21.557770 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9f55ec8e8d47f3959084440c64754b" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7"
I20260929 03:22:21.557796 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9f55ec8e8d47f3959084440c64754b" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afd1fd787764319b97f3729bd920eaa"
I20260929 03:22:21.557868 29322 raft_consensus.cc:3037] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.557866 29194 raft_consensus.cc:3037] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.558583 29194 raft_consensus.cc:2454] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 1.
I20260929 03:22:21.558611 29322 raft_consensus.cc:2454] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 1.
I20260929 03:22:21.558785 29408 leader_election.cc:302] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dfb0c040964dab86428a4fd093d2f7, c4205ebd0af04d568f47a5770eabeef0; no voters:
I20260929 03:22:21.558902 29549 raft_consensus.cc:2781] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:21.559105 29549 raft_consensus.cc:686] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 1 LEADER]: Becoming Leader. State: Replica: c4205ebd0af04d568f47a5770eabeef0, State: Running, Role: LEADER
I20260929 03:22:21.559245 29549 consensus_queue.cc:237] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } }
I20260929 03:22:21.560060 29036 catalog_manager.cc:5495] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67). New cstate: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:21.588506 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.585809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.27.68.67:34789)
0929 03:22:21.585877 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.585879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.585889 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:21.585967 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.585968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.585995 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.586072 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:21.586243 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.586772 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.587610 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.587810 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.587890 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.588113 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:21.588224 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:21.588320 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.588367 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.588390 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:22:21.590610 29322 raft_consensus.cc:1261] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Refusing update from remote peer c4205ebd0af04d568f47a5770eabeef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:21.590663 29194 raft_consensus.cc:1261] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Refusing update from remote peer c4205ebd0af04d568f47a5770eabeef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:21.590997 29554 consensus_queue.cc:1035] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:21.591135 29549 consensus_queue.cc:1035] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:21.595281 29559 mvcc.cc:205] Tried to move back new op lower bound from 7334511171952394240 to 7334511171827474432. Current Snapshot: MvccSnapshot[applied={T|T < 7334511171952394240}]
I20260929 03:22:21.597193 29035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:21.597281 29035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:21.598050 29035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Sending DeleteTablet for 3 replicas of tablet 2f9f55ec8e8d47f3959084440c64754b
I20260929 03:22:21.598372 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9f55ec8e8d47f3959084440c64754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:21 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:42806
I20260929 03:22:21.598408 29314 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9f55ec8e8d47f3959084440c64754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:21 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:33136
I20260929 03:22:21.598474 29174 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9f55ec8e8d47f3959084440c64754b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:21 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:47016
I20260929 03:22:21.598692 29035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:22:21.598753 29567 tablet_replica.cc:330] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: stopping tablet replica
W20260929 03:22:21.598841 29035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:21.598855 29567 raft_consensus.cc:2229] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:21.598944 29567 raft_consensus.cc:2258] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:21.599714 29567 ts_tablet_manager.cc:1905] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:21.600208 29454 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 (DEFAULT_TABLE table=test-table [id=45294ff36a5045ee8e3d4310d42f2220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.600247 29174 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 (DEFAULT_TABLE table=test-table [id=45294ff36a5045ee8e3d4310d42f2220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.600322 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8b55128dc4a4f93f741d1787102c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.600345 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8b55128dc4a4f93f741d1787102c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.600521 29314 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 (DEFAULT_TABLE table=test-table [id=45294ff36a5045ee8e3d4310d42f2220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:21.600656 29314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8b55128dc4a4f93f741d1787102c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:21.601495 29567 ts_tablet_manager.cc:1918] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:21.601563 29567 log.cc:1196] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/2f9f55ec8e8d47f3959084440c64754b
I20260929 03:22:21.601686 29544 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Bootstrap starting.
I20260929 03:22:21.602063 29545 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Bootstrap starting.
I20260929 03:22:21.602087 29544 tablet_bootstrap.cc:654] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.602424 29548 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Bootstrap starting.
I20260929 03:22:21.602492 29545 tablet_bootstrap.cc:654] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.602782 29548 tablet_bootstrap.cc:654] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:21.603297 29544 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: No bootstrap required, opened a new log
I20260929 03:22:21.603351 29544 ts_tablet_manager.cc:1397] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:21.603349 29568 tablet_replica.cc:330] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: stopping tablet replica
I20260929 03:22:21.603459 29568 raft_consensus.cc:2229] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:21.603490 29544 raft_consensus.cc:348] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.603574 29544 raft_consensus.cc:374] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.603588 29568 raft_consensus.cc:2258] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:21.603601 29544 raft_consensus.cc:729] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4205ebd0af04d568f47a5770eabeef0, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.603648 29544 consensus_queue.cc:260] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.603745 29569 tablet_replica.cc:330] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: stopping tablet replica
I20260929 03:22:21.603749 29544 ts_tablet_manager.cc:1428] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.603852 29569 raft_consensus.cc:2229] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:21.603966 29569 raft_consensus.cc:2258] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:21.604112 29568 ts_tablet_manager.cc:1905] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:21.603842 29548 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: No bootstrap required, opened a new log
I20260929 03:22:21.604318 29548 ts_tablet_manager.cc:1397] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:21.604422 29569 ts_tablet_manager.cc:1905] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:21.604462 29548 raft_consensus.cc:348] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.604534 29548 raft_consensus.cc:374] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.604557 29548 raft_consensus.cc:729] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afd1fd787764319b97f3729bd920eaa, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.604601 29567 ts_tablet_manager.cc:1939] T 2f9f55ec8e8d47f3959084440c64754b P c4205ebd0af04d568f47a5770eabeef0: Deleting consensus metadata
I20260929 03:22:21.604604 29548 consensus_queue.cc:260] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.604713 29548 ts_tablet_manager.cc:1428] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:21.605023 29022 catalog_manager.cc:4841] TS c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67:34789): tablet 2f9f55ec8e8d47f3959084440c64754b (table test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]) successfully deleted
I20260929 03:22:21.605427 29545 tablet_bootstrap.cc:492] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: No bootstrap required, opened a new log
I20260929 03:22:21.605480 29545 ts_tablet_manager.cc:1397] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:22:21.605628 29545 raft_consensus.cc:348] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.605719 29545 raft_consensus.cc:374] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:21.605747 29545 raft_consensus.cc:729] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dfb0c040964dab86428a4fd093d2f7, State: Initialized, Role: FOLLOWER
I20260929 03:22:21.605798 29545 consensus_queue.cc:260] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.605921 29545 ts_tablet_manager.cc:1428] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:21.605787 29568 ts_tablet_manager.cc:1918] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:21.605997 29569 ts_tablet_manager.cc:1918] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:21.606024 29568 log.cc:1196] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/2f9f55ec8e8d47f3959084440c64754b
I20260929 03:22:21.606071 29569 log.cc:1196] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/2f9f55ec8e8d47f3959084440c64754b
I20260929 03:22:21.606310 29568 ts_tablet_manager.cc:1939] T 2f9f55ec8e8d47f3959084440c64754b P 64dfb0c040964dab86428a4fd093d2f7: Deleting consensus metadata
I20260929 03:22:21.606405 29569 ts_tablet_manager.cc:1939] T 2f9f55ec8e8d47f3959084440c64754b P 9afd1fd787764319b97f3729bd920eaa: Deleting consensus metadata
I20260929 03:22:21.606870 29023 catalog_manager.cc:4841] TS 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525): tablet 2f9f55ec8e8d47f3959084440c64754b (table test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]) successfully deleted
I20260929 03:22:21.606976 29023 catalog_manager.cc:4841] TS 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419): tablet 2f9f55ec8e8d47f3959084440c64754b (table test-table [id=a5e20cccaf804a12ad2ecb8505d0f2e2]) successfully deleted
W20260929 03:22:21.681198 29521 tablet.cc:2369] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:22:21.733050 29241 tablet.cc:2369] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:21.983233 29552 raft_consensus.cc:482] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:21.983325 29552 raft_consensus.cc:504] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.983619 29552 leader_election.cc:288] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419), c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67:34789)
I20260929 03:22:21.987675 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.983934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:50589 (local address 127.27.68.67:34789)
0929 03:22:21.983980 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.983982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.983993 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:21.984095 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.984096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.984111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.984170 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.984343 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.985090 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.986725 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.987344 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.987407 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.987415 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.987563 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.987608 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:22:21.987696 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.983833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34789 (local address 127.27.68.65:50589)
0929 03:22:21.983943 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.983953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.984007 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.984174 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.984174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.984176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.984243 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.984246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.985191 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.985193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.986635 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.986638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.987428 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.987429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.987495 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.987499 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.987553 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.987598 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":45,"thread_start_us":43,"threads_started":1}
I20260929 03:22:21.987797 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.983877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:59419 (local address 127.27.68.66:39419)
0929 03:22:21.983923 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:21.983926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:21.984008 (+ 82us) server_negotiation.cc:407] Connection header received
0929 03:22:21.984095 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:21.984096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:21.984121 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:21.984170 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:21.984335 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.985178 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.986743 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.987383 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.987461 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.987469 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:21.987662 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 03:22:21.987712 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:21.988039 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:21.983821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39419 (local address 127.27.68.65:59419)
0929 03:22:21.983940 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:21.983953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:21.984024 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:21.984216 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:21.984217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:21.984218 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:21.984256 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.984257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.985288 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.985289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.986635 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:21.986638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:21.987477 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:21.987478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:21.987569 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:21.987574 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:21.987873 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 03:22:21.987926 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":69,"thread_start_us":62,"threads_started":1}
I20260929 03:22:21.988128 29468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8b55128dc4a4f93f741d1787102c7" candidate_uuid: "9afd1fd787764319b97f3729bd920eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4205ebd0af04d568f47a5770eabeef0" is_pre_election: true
I20260929 03:22:21.988281 29468 raft_consensus.cc:2454] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afd1fd787764319b97f3729bd920eaa in term 0.
I20260929 03:22:21.988343 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8b55128dc4a4f93f741d1787102c7" candidate_uuid: "9afd1fd787764319b97f3729bd920eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7" is_pre_election: true
I20260929 03:22:21.988430 29322 raft_consensus.cc:2454] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afd1fd787764319b97f3729bd920eaa in term 0.
I20260929 03:22:21.988611 29127 leader_election.cc:302] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dfb0c040964dab86428a4fd093d2f7, 9afd1fd787764319b97f3729bd920eaa; no voters:
I20260929 03:22:21.988757 29552 raft_consensus.cc:2781] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:21.988833 29552 raft_consensus.cc:482] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:21.988866 29552 raft_consensus.cc:3037] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.989553 29552 raft_consensus.cc:504] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.989717 29552 leader_election.cc:288] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [CANDIDATE]: Term 1 election: Requested vote from peers 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419), c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67:34789)
I20260929 03:22:21.989851 29468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8b55128dc4a4f93f741d1787102c7" candidate_uuid: "9afd1fd787764319b97f3729bd920eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4205ebd0af04d568f47a5770eabeef0"
I20260929 03:22:21.989850 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8b55128dc4a4f93f741d1787102c7" candidate_uuid: "9afd1fd787764319b97f3729bd920eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7"
I20260929 03:22:21.989928 29322 raft_consensus.cc:3037] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.989928 29468 raft_consensus.cc:3037] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:21.990675 29322 raft_consensus.cc:2454] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afd1fd787764319b97f3729bd920eaa in term 1.
I20260929 03:22:21.990787 29468 raft_consensus.cc:2454] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afd1fd787764319b97f3729bd920eaa in term 1.
I20260929 03:22:21.990854 29127 leader_election.cc:302] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dfb0c040964dab86428a4fd093d2f7, 9afd1fd787764319b97f3729bd920eaa; no voters:
I20260929 03:22:21.990952 29552 raft_consensus.cc:2781] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:21.991129 29552 raft_consensus.cc:686] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 1 LEADER]: Becoming Leader. State: Replica: 9afd1fd787764319b97f3729bd920eaa, State: Running, Role: LEADER
I20260929 03:22:21.991221 29552 consensus_queue.cc:237] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:21.991971 29036 catalog_manager.cc:5495] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9afd1fd787764319b97f3729bd920eaa (127.27.68.65). New cstate: current_term: 1 leader_uuid: "9afd1fd787764319b97f3729bd920eaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:22:22 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:22.067559 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.060050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.27.68.126:46697)
0929 03:22:22.060212 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.060214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.060224 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:22.060284 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.060284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.060294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.060356 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:22.060825 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.061334 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.062031 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.062233 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.062298 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.064606 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:22.064608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:22.064608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:22.064616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:22.066184 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.066614 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.066615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.066616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.066679 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.066879 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.066880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.066881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.067027 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:22.067101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.067322 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.067437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260929 03:22:22.067626 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.059965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46697 (local address 127.0.0.1:39860)
0929 03:22:22.060134 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:22.060147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:22.060229 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:22.060374 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:22.060375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:22.060686 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:22.060749 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.060751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.061408 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.061408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.061940 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.061942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.062306 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.062307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.062346 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.062937 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:22.062941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:22.064503 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:22.066292 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:22.066303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:22.066306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:22.066525 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:22.066744 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:22.066745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:22.066745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:22.066817 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:22.067103 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:22.067114 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:22.067162 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:22:22.067294 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:22:22.067503 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":95,"thread_start_us":86,"threads_started":1}
I20260929 03:22:22.069576 29036 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:39860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:22:22.069793 29036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:22.071373 29174 tablet_service.cc:1467] Processing CreateTablet for tablet 63525a2aa03d44c997a04aede665b45f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a9fd5b79204ce0bb4263569fa34848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:22.071393 29314 tablet_service.cc:1467] Processing CreateTablet for tablet 63525a2aa03d44c997a04aede665b45f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a9fd5b79204ce0bb4263569fa34848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:22.071551 29174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63525a2aa03d44c997a04aede665b45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:22.071554 29314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63525a2aa03d44c997a04aede665b45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:22.072818 29545 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: Bootstrap starting.
I20260929 03:22:22.072896 29548 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: Bootstrap starting.
I20260929 03:22:22.072999 29454 tablet_service.cc:1467] Processing CreateTablet for tablet 63525a2aa03d44c997a04aede665b45f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a9fd5b79204ce0bb4263569fa34848]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:22.073166 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63525a2aa03d44c997a04aede665b45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:22.073302 29548 tablet_bootstrap.cc:654] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:22.073297 29545 tablet_bootstrap.cc:654] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:22.073930 29545 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: No bootstrap required, opened a new log
I20260929 03:22:22.074000 29545 ts_tablet_manager.cc:1397] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:22.074185 29545 raft_consensus.cc:348] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.074270 29545 raft_consensus.cc:374] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:22.074291 29545 raft_consensus.cc:729] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dfb0c040964dab86428a4fd093d2f7, State: Initialized, Role: FOLLOWER
I20260929 03:22:22.074301 29548 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: No bootstrap required, opened a new log
I20260929 03:22:22.074342 29548 ts_tablet_manager.cc:1397] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:22.074337 29545 consensus_queue.cc:260] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.074450 29545 ts_tablet_manager.cc:1428] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:22.074501 29548 raft_consensus.cc:348] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.074579 29548 raft_consensus.cc:374] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:22.074604 29548 raft_consensus.cc:729] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afd1fd787764319b97f3729bd920eaa, State: Initialized, Role: FOLLOWER
I20260929 03:22:22.074651 29548 consensus_queue.cc:260] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.074606 29550 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.074704 29550 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.074741 29548 ts_tablet_manager.cc:1428] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:22.074848 29544 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: Bootstrap starting.
I20260929 03:22:22.074842 29552 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.074906 29552 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.075304 29544 tablet_bootstrap.cc:654] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:22.075847 29544 tablet_bootstrap.cc:492] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: No bootstrap required, opened a new log
I20260929 03:22:22.075911 29544 ts_tablet_manager.cc:1397] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:22.076014 29544 raft_consensus.cc:348] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.076097 29544 raft_consensus.cc:374] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:22.076126 29544 raft_consensus.cc:729] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4205ebd0af04d568f47a5770eabeef0, State: Initialized, Role: FOLLOWER
I20260929 03:22:22.076179 29544 consensus_queue.cc:260] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.076282 29544 ts_tablet_manager.cc:1428] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:22.076319 29564 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.076438 29564 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.096416 29564 raft_consensus.cc:482] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:22.096510 29564 raft_consensus.cc:504] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.096681 29564 leader_election.cc:288] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525), 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419)
I20260929 03:22:22.096931 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63525a2aa03d44c997a04aede665b45f" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afd1fd787764319b97f3729bd920eaa" is_pre_election: true
I20260929 03:22:22.097016 29194 raft_consensus.cc:2454] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 0.
I20260929 03:22:22.096953 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63525a2aa03d44c997a04aede665b45f" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7" is_pre_election: true
I20260929 03:22:22.097092 29322 raft_consensus.cc:2454] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 0.
I20260929 03:22:22.097222 29408 leader_election.cc:302] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afd1fd787764319b97f3729bd920eaa, c4205ebd0af04d568f47a5770eabeef0; no voters:
I20260929 03:22:22.097337 29564 raft_consensus.cc:2781] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:22.097380 29564 raft_consensus.cc:482] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:22.097400 29564 raft_consensus.cc:3037] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:22.098111 29564 raft_consensus.cc:504] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.098250 29564 leader_election.cc:288] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 election: Requested vote from peers 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525), 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419)
I20260929 03:22:22.098399 29194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63525a2aa03d44c997a04aede665b45f" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afd1fd787764319b97f3729bd920eaa"
I20260929 03:22:22.098421 29322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63525a2aa03d44c997a04aede665b45f" candidate_uuid: "c4205ebd0af04d568f47a5770eabeef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dfb0c040964dab86428a4fd093d2f7"
I20260929 03:22:22.098487 29322 raft_consensus.cc:3037] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:22.098512 29194 raft_consensus.cc:3037] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:22.099128 29322 raft_consensus.cc:2454] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 1.
I20260929 03:22:22.099207 29194 raft_consensus.cc:2454] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4205ebd0af04d568f47a5770eabeef0 in term 1.
I20260929 03:22:22.099335 29408 leader_election.cc:302] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dfb0c040964dab86428a4fd093d2f7, c4205ebd0af04d568f47a5770eabeef0; no voters:
I20260929 03:22:22.099419 29564 raft_consensus.cc:2781] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:22.099540 29564 raft_consensus.cc:686] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [term 1 LEADER]: Becoming Leader. State: Replica: c4205ebd0af04d568f47a5770eabeef0, State: Running, Role: LEADER
I20260929 03:22:22.099597 29564 consensus_queue.cc:237] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } }
I20260929 03:22:22.099866 29587 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: TxnStatusTablet state changed. Reason: New leader c4205ebd0af04d568f47a5770eabeef0. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.099964 29587 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:22.100149 29589 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:22:22.100277 29589 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:22:22.100196 29035 catalog_manager.cc:5495] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67). New cstate: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } health_report { overall_health: HEALTHY } } }
I20260929 03:22:22.113456 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.111101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.27.68.67:34789)
0929 03:22:22.111160 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.111162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.111176 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:22:22.111197 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.111197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.111205 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.111302 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:22.111464 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.111935 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.112728 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.112920 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.113004 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.113175 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.113212 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.113271 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.113344 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.113367 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:22.113462 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.110871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34789 (local address 127.0.0.1:42826)
0929 03:22:22.110953 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:22.110965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:22.111021 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:22.111330 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:22.111332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:22.111333 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:22.111387 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.111389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.112037 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.112039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.112637 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.112639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.112994 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.112995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.113052 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.113071 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.113283 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.113292 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:22:22.113342 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:22:22.113395 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:22:22.114504 29322 raft_consensus.cc:1261] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Refusing update from remote peer c4205ebd0af04d568f47a5770eabeef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:22.114698 29194 raft_consensus.cc:1261] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Refusing update from remote peer c4205ebd0af04d568f47a5770eabeef0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:22.114814 29564 consensus_queue.cc:1035] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:22.114950 29587 consensus_queue.cc:1035] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:22.115450 29550 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: TxnStatusTablet state changed. Reason: New leader c4205ebd0af04d568f47a5770eabeef0. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.115542 29550 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.116082 29552 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: TxnStatusTablet state changed. Reason: New leader c4205ebd0af04d568f47a5770eabeef0. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.116174 29552 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.116539 29587 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.116631 29587 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:22.116889 29587 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.116950 29587 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P c4205ebd0af04d568f47a5770eabeef0: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:22.117027 29550 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.117113 29550 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 64dfb0c040964dab86428a4fd093d2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.117408 29552 tablet_replica.cc:439] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4205ebd0af04d568f47a5770eabeef0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9afd1fd787764319b97f3729bd920eaa" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37525 } } peers { permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 } } peers { permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 } } }
I20260929 03:22:22.117482 29552 tablet_replica.cc:442] T 63525a2aa03d44c997a04aede665b45f P 9afd1fd787764319b97f3729bd920eaa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:22.117899 29595 mvcc.cc:205] Tried to move back new op lower bound from 7334511174099247104 to 7334511174040375296. Current Snapshot: MvccSnapshot[applied={T|T < 7334511174099247104 or (T in {7334511174099247104})}]
I20260929 03:22:22.122011 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.119369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.27.68.65:37525)
0929 03:22:22.119477 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.119479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.119487 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:22.119575 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.119576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.119596 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.119696 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:22.119825 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.120334 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.121108 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.121336 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.121420 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.121602 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.121723 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.121810 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.121849 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.121870 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260929 03:22:22.127127 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.124722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:50703 (local address 127.27.68.67:34789)
0929 03:22:22.124775 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.124777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.124849 (+ 72us) server_negotiation.cc:407] Connection header received
0929 03:22:22.124916 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.124917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.124930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.124985 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:22.125139 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.125606 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.126386 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.126599 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.126685 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.126857 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.126890 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.126957 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.127026 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.127047 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:22:22.127149 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.124652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34789 (local address 127.27.68.65:50703)
0929 03:22:22.124796 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:22.124807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:22.124866 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:22.124989 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:22.124990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:22.124991 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:22.125062 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.125063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.125698 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.125699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.126300 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.126301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.126687 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.126688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.126752 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.126776 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.126959 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.126962 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:22.127021 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:22:22.127067 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260929 03:22:22.129355 29322 raft_consensus.cc:1261] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Refusing update from remote peer 9afd1fd787764319b97f3729bd920eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:22.129456 29468 raft_consensus.cc:1261] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Refusing update from remote peer 9afd1fd787764319b97f3729bd920eaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:22.129604 29575 consensus_queue.cc:1035] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dfb0c040964dab86428a4fd093d2f7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:22.129916 29575 consensus_queue.cc:1035] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4205ebd0af04d568f47a5770eabeef0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:22.132462 29592 mvcc.cc:205] Tried to move back new op lower bound from 7334511174159884288 to 7334511173596835840. Current Snapshot: MvccSnapshot[applied={T|T < 7334511174159884288}]
I20260929 03:22:22.138212 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.135511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37525 (local address 127.27.68.67:33857)
0929 03:22:22.135681 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:22.135691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:22.135779 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:22.135897 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:22.135898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:22.135900 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:22.135947 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.135948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.136568 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.136569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.137437 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:22.137440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.137787 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.137788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:22.137840 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.137855 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.138078 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.138080 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:22.138112 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:22:22.138147 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260929 03:22:22.138219 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.135570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:33857 (local address 127.27.68.65:37525)
0929 03:22:22.135617 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.135619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.135764 (+ 145us) server_negotiation.cc:407] Connection header received
0929 03:22:22.135831 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.135832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.135843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.135886 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:22.135992 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.136478 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.137516 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.137717 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.137789 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.137969 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:22.138010 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:22.138069 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.138124 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.138149 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:22:22.148046 29036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:22.148180 29036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:22.148823 29036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9307d90bea649b0b1e343db7e9fae4f: Sending DeleteTablet for 3 replicas of tablet 0db8b55128dc4a4f93f741d1787102c7
I20260929 03:22:22.149120 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:22 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:42806
I20260929 03:22:22.149164 29314 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:22 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:33136
I20260929 03:22:22.149984 29174 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8b55128dc4a4f93f741d1787102c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:22 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:47016
I20260929 03:22:22.150282 29617 tablet_replica.cc:330] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: stopping tablet replica
I20260929 03:22:22.150349 29617 raft_consensus.cc:2229] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:22.150457 29617 raft_consensus.cc:2258] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:22.150758 29617 ts_tablet_manager.cc:1905] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:22.152757 29614 tablet_replica.cc:330] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: stopping tablet replica
I20260929 03:22:22.152834 29614 raft_consensus.cc:2229] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:22.152912 29614 raft_consensus.cc:2258] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:22.153064 29615 tablet_replica.cc:330] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: stopping tablet replica
I20260929 03:22:22.153131 29615 raft_consensus.cc:2229] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:22.153178 29615 raft_consensus.cc:2258] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:22.153232 29614 ts_tablet_manager.cc:1905] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:22.153469 29615 ts_tablet_manager.cc:1905] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:22.154795 29614 ts_tablet_manager.cc:1918] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:22.154816 29617 ts_tablet_manager.cc:1918] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:22.154860 29614 log.cc:1196] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/0db8b55128dc4a4f93f741d1787102c7
I20260929 03:22:22.154867 29617 log.cc:1196] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/0db8b55128dc4a4f93f741d1787102c7
I20260929 03:22:22.155097 29617 ts_tablet_manager.cc:1939] T 0db8b55128dc4a4f93f741d1787102c7 P 9afd1fd787764319b97f3729bd920eaa: Deleting consensus metadata
I20260929 03:22:22.155097 29614 ts_tablet_manager.cc:1939] T 0db8b55128dc4a4f93f741d1787102c7 P c4205ebd0af04d568f47a5770eabeef0: Deleting consensus metadata
I20260929 03:22:22.155278 29615 ts_tablet_manager.cc:1918] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:22.155341 29615 log.cc:1196] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/0db8b55128dc4a4f93f741d1787102c7
I20260929 03:22:22.155443 29023 catalog_manager.cc:4841] TS 9afd1fd787764319b97f3729bd920eaa (127.27.68.65:37525): tablet 0db8b55128dc4a4f93f741d1787102c7 (table test-table [id=45294ff36a5045ee8e3d4310d42f2220]) successfully deleted
I20260929 03:22:22.155443 29022 catalog_manager.cc:4841] TS c4205ebd0af04d568f47a5770eabeef0 (127.27.68.67:34789): tablet 0db8b55128dc4a4f93f741d1787102c7 (table test-table [id=45294ff36a5045ee8e3d4310d42f2220]) successfully deleted
I20260929 03:22:22.155576 29615 ts_tablet_manager.cc:1939] T 0db8b55128dc4a4f93f741d1787102c7 P 64dfb0c040964dab86428a4fd093d2f7: Deleting consensus metadata
I20260929 03:22:22.155900 29023 catalog_manager.cc:4841] TS 64dfb0c040964dab86428a4fd093d2f7 (127.27.68.66:39419): tablet 0db8b55128dc4a4f93f741d1787102c7 (table test-table [id=45294ff36a5045ee8e3d4310d42f2220]) successfully deleted
Sep 29 03:22:22 dist-test-slave-fzwk krb5kdc[28991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:22.160987 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.150180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.27.68.65:37525)
0929 03:22:22.150221 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.150223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.150395 (+ 172us) server_negotiation.cc:407] Connection header received
0929 03:22:22.150607 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.150608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.150620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.150649 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:22.151681 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.152396 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.153875 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.154135 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.154180 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.157904 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:22.157914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:22.157914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:22.157923 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:22.159728 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.160110 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.160118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.160119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.160193 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.160396 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.160397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.160398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.160556 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:22.160711 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.160777 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.160865 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:22:22.161362 29234 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47044
I20260929 03:22:22.168787 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:22.162319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.27.68.126:46697)
0929 03:22:22.162365 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:22.162368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:22.162529 (+ 161us) server_negotiation.cc:407] Connection header received
0929 03:22:22.162597 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:22.162598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:22.162609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:22.162654 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:22.163262 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.163783 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.164560 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:22.164785 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:22.164863 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:22.166098 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:22.166100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:22.166100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:22.166108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:22.167626 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.167989 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.167991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.167992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.168053 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:22.168251 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:22.168252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:22.168253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:22.168374 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:22.168482 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:22.168592 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:22:22.168682 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:22:22.169126 29036 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39864
I20260929 03:22:22.169750 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29108
I20260929 03:22:22.176097 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29249
I20260929 03:22:22.183128 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29389
I20260929 03:22:22.188874 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29004
2026-09-29T03:22:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6046 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:22 dist-test-slave-fzwk krb5kdc[29623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:22 dist-test-slave-fzwk krb5kdc[29623](info): set up 2 sockets
Sep 29 03:22:22 dist-test-slave-fzwk krb5kdc[29623](info): commencing operation
krb5kdc: starting...
W20260929 03:22:23.211695 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:23.331419 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44103
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:23.416081 29636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:23.416373 29636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:23.416401 29636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:23.417610 29636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:23.417654 29636 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:22:23.417667 29636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:23.417680 29636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:23.417732 29636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:23.417750 29636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:23.419278 29636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44103
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:23.419548 29636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:23.419797 29636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:23.424604 29642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.424643 29641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.424608 29644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:23.424993 29636 server_base.cc:1034] running on GCE node
I20260929 03:22:23.425180 29636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:23.425465 29636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:23.426618 29636 hybrid_clock.cc:648] HybridClock initialized: now 1790652143426607 us; error 27 us; skew 500 ppm
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:23.429230 29636 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:23.429733 29636 webserver.cc:460] Webserver started at http://127.27.68.126:34281/ using document root <none> and password file <none>
I20260929 03:22:23.429983 29636 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:23.430058 29636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:23.430208 29636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:23.431353 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "f6a8354a35af4fb5acffea4367e5ec17"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "dc76184e48c20453c91823c8a28deb1b"
server_key_iv: "081888822d16cd2e171b52fb4623a2de"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.431728 29636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "f6a8354a35af4fb5acffea4367e5ec17"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "dc76184e48c20453c91823c8a28deb1b"
server_key_iv: "081888822d16cd2e171b52fb4623a2de"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.433326 29636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:22:23.434082 29651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.434260 29636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:23.434368 29636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "f6a8354a35af4fb5acffea4367e5ec17"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "dc76184e48c20453c91823c8a28deb1b"
server_key_iv: "081888822d16cd2e171b52fb4623a2de"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.434445 29636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:23.445451 29636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:23.448037 29636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:23.448213 29636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:23.453140 29636 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44103
I20260929 03:22:23.453191 29713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44103 every 8 connection(s)
I20260929 03:22:23.453557 29636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:23.454195 29714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:23.456692 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Bootstrap starting.
I20260929 03:22:23.456960 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29636
I20260929 03:22:23.457060 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:23.457223 27921 external_mini_cluster.cc:1428] Setting key f65c326462e82e79e33209e288a7c131
I20260929 03:22:23.457443 29714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:23.457806 29714 log.cc:824] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:23.458542 29714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: No bootstrap required, opened a new log
I20260929 03:22:23.460026 29714 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } }
I20260929 03:22:23.460167 29714 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:23.460191 29714 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a8354a35af4fb5acffea4367e5ec17, State: Initialized, Role: FOLLOWER
I20260929 03:22:23.460350 29714 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } }
I20260929 03:22:23.460425 29714 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:23.460448 29714 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:23.460520 29714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:23.461089 29714 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } }
I20260929 03:22:23.461221 29714 leader_election.cc:302] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6a8354a35af4fb5acffea4367e5ec17; no voters:
I20260929 03:22:23.461427 29714 leader_election.cc:288] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:23.461452 29719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:23.461561 29719 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [term 1 LEADER]: Becoming Leader. State: Replica: f6a8354a35af4fb5acffea4367e5ec17, State: Running, Role: LEADER
I20260929 03:22:23.461648 29719 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } }
I20260929 03:22:23.461696 29714 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:23.461973 29721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6a8354a35af4fb5acffea4367e5ec17. Latest consensus state: current_term: 1 leader_uuid: "f6a8354a35af4fb5acffea4367e5ec17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } } }
I20260929 03:22:23.462070 29721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:23.462219 29720 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6a8354a35af4fb5acffea4367e5ec17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6a8354a35af4fb5acffea4367e5ec17" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44103 } } }
I20260929 03:22:23.462270 29720 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:23.462378 29724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:23.462867 29724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:23.465953 29724 catalog_manager.cc:1347] Generated new cluster ID: 7bf7effa510e4520a8658e0fcb8aa091
I20260929 03:22:23.466301 29724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:23.470196 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.457784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.27.68.126:44103)
0929 03:22:23.457982 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.457992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.458039 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:22:23.458494 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.458501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.458555 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.458725 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.459397 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.460922 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.461951 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.462617 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.462729 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.465715 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.465721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.465721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.465734 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.467892 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.468344 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.468348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.468348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.468471 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.468682 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.468683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.468683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.469031 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.469712 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.469796 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.469962 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260929 03:22:23.476430 29724 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:23.477272 29724 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:23.484122 29724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Generated new TSK 0
I20260929 03:22:23.484355 29724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:23.513876 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44103
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:23.597373 29740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:23.597594 29740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:23.597621 29740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:23.598897 29740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:23.598942 29740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:23.598964 29740 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:22:23.598976 29740 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:23.598994 29740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:23.600521 29740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:23.600800 29740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:23.601047 29740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:23.606123 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.606175 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.606123 29748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:23.606248 29740 server_base.cc:1034] running on GCE node
I20260929 03:22:23.606575 29740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:23.606840 29740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:23.607987 29740 hybrid_clock.cc:648] HybridClock initialized: now 1790652143607975 us; error 29 us; skew 500 ppm
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:23.610450 29740 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:23.610890 29740 webserver.cc:460] Webserver started at http://127.27.68.65:38827/ using document root <none> and password file <none>
I20260929 03:22:23.611125 29740 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:23.611202 29740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:23.611400 29740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:23.612509 29740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "d1b085ddb0bf4ac5abc23c527e22bce7"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "2590b3db7b93aceddf18361b01409857"
server_key_iv: "59dccfaa249dec9acca2df4470dc5275"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.612890 29740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "d1b085ddb0bf4ac5abc23c527e22bce7"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "2590b3db7b93aceddf18361b01409857"
server_key_iv: "59dccfaa249dec9acca2df4470dc5275"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.614236 29740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:23.614928 29755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.615095 29740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:23.615195 29740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "d1b085ddb0bf4ac5abc23c527e22bce7"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "2590b3db7b93aceddf18361b01409857"
server_key_iv: "59dccfaa249dec9acca2df4470dc5275"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.615330 29740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:23.642467 29740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:23.644423 29740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:23.644603 29740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:23.645167 29740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:23.645237 29740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.645278 29740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:23.645323 29740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.652200 29740 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:40861
I20260929 03:22:23.652289 29870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:40861 every 8 connection(s)
I20260929 03:22:23.652635 29740 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:23.655694 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.645679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:47445 (local address 127.27.68.126:44103)
0929 03:22:23.645751 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.645754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.645993 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:22:23.646492 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.646494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.646510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.646599 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.647530 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.648394 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.650008 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.650362 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.650462 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.653258 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.653259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.653259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.653267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.654426 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.654806 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.654807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.654807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.654859 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.655067 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.655067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.655067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.655176 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.655296 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.655541 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.655628 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:22:23.655875 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.645676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.65:47445)
0929 03:22:23.645848 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.645891 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.646388 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.646698 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.646702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.647097 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.647420 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.647427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.648485 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.648486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.649896 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.649901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.650464 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.650465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.650571 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.651281 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.651292 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.653156 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.654500 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.654507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.654509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.654735 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.654916 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.654917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.654917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.655014 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.655289 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.655295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.655426 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.655514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.655640 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260929 03:22:23.656392 29871 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44103
I20260929 03:22:23.656481 29871 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:23.656684 29871 heartbeater.cc:511] Master 127.27.68.126:44103 requested a full tablet report, sending...
I20260929 03:22:23.657181 29667 ts_manager.cc:194] Registered new tserver with Master: d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861)
I20260929 03:22:23.658025 29667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:47445
I20260929 03:22:23.659556 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29740
I20260929 03:22:23.659651 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:23.659876 27921 external_mini_cluster.cc:1428] Setting key 0fba99f151b986c7f5321c312b6ab27d
I20260929 03:22:23.663357 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.657481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.65:46265)
0929 03:22:23.657602 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.657613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.657678 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.657799 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.657800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.657953 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.658006 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.658008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.658738 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.658739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.659341 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.659343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.659835 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.659836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.659894 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.660381 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.660386 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.660736 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.662239 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.662240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.662240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.662472 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.662668 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.662668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.662669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.662751 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.662987 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.662988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.663035 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.663120 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.663255 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260929 03:22:23.663350 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.657544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46265 (local address 127.27.68.126:44103)
0929 03:22:23.657586 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.657588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.657683 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:22:23.657729 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.657730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.657739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.657787 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.658076 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.658645 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.659446 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.659749 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.659826 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.660815 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.660817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.660817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.660825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.662140 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.662534 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.662536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.662536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.662598 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.662800 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.662801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.662802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.662915 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.662986 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.663156 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.663256 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:23.685750 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44103
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:23.768436 29880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:23.768656 29880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:23.768682 29880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:23.769879 29880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:23.769925 29880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:23.769948 29880 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:22:23.769959 29880 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:23.769977 29880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:23.771507 29880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:23.771780 29880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:23.772019 29880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:23.777000 29888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.777015 29886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:23.777215 29880 server_base.cc:1034] running on GCE node
W20260929 03:22:23.777010 29885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:23.777591 29880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:23.777841 29880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:23.778999 29880 hybrid_clock.cc:648] HybridClock initialized: now 1790652143778984 us; error 31 us; skew 500 ppm
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:23.781622 29880 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:23.782068 29880 webserver.cc:460] Webserver started at http://127.27.68.66:39581/ using document root <none> and password file <none>
I20260929 03:22:23.782265 29880 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:23.782310 29880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:23.782404 29880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:23.783596 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "d7bc6b0cc3bc6e4337f22fc0e182f98d"
server_key_iv: "b8a8a8b01c991fe70532eaa19ed7cc92"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.783917 29880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "d7bc6b0cc3bc6e4337f22fc0e182f98d"
server_key_iv: "b8a8a8b01c991fe70532eaa19ed7cc92"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.785254 29880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:23.785986 29895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.786182 29880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:23.786281 29880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "d7bc6b0cc3bc6e4337f22fc0e182f98d"
server_key_iv: "b8a8a8b01c991fe70532eaa19ed7cc92"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.786340 29880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:23.800329 29880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:23.802223 29880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:23.802336 29880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:23.802877 29880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:23.802919 29880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.802942 29880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:23.802977 29880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.809947 29880 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:43575
I20260929 03:22:23.810009 30010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:43575 every 8 connection(s)
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:23.810344 29880 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:23.810776 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 29880
I20260929 03:22:23.810878 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:23.811086 27921 external_mini_cluster.cc:1428] Setting key fd964126e99644691dd805eacba8d3a7
I20260929 03:22:23.814556 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.803403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51929 (local address 127.27.68.126:44103)
0929 03:22:23.803447 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.803449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.803694 (+ 245us) server_negotiation.cc:407] Connection header received
0929 03:22:23.804233 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.804235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.804247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.804304 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.805214 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.806077 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.807790 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.808107 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.808192 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.811113 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.811115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.811115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.811126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.813078 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.813495 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.813497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.813498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.813567 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.813862 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.813863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.813864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.813995 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.814088 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.814367 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.814456 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:22:23.814791 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.803402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.66:51929)
0929 03:22:23.803567 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.803604 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.804124 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.804379 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.804383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.804787 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.805103 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.805109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.806212 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.806214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.807672 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.807676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.808192 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.808192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.808303 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.808893 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.808905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.810826 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.813167 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.813172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.813174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.813405 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.813637 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.813638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.813638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.813790 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.814080 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.814086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.814234 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.814348 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.814519 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260929 03:22:23.815304 30011 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44103
I20260929 03:22:23.815392 30011 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:23.815603 30011 heartbeater.cc:511] Master 127.27.68.126:44103 requested a full tablet report, sending...
I20260929 03:22:23.815974 29668 ts_manager.cc:194] Registered new tserver with Master: 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575)
I20260929 03:22:23.816423 29668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:51929
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
I20260929 03:22:23.822434 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.816520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:49397 (local address 127.27.68.126:44103)
0929 03:22:23.816558 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.816560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.816569 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:23.816664 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.816665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.816677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.816752 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.817143 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.817973 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.818668 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.818992 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.819069 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.820136 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.820137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.820138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.820146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.821435 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.821752 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.821753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.821753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.821801 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.821956 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.821956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.821956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.822117 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.822167 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.822314 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.822372 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:23.822438 30018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.816357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.66:49397)
0929 03:22:23.816505 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.816517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.816572 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.816762 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.816763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.816981 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.817059 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.817061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.818083 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.818084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.818585 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.818587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.819103 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.819104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.819185 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.819719 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.819724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.820041 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.821502 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.821503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.821503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.821683 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.821841 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.821841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.821842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.821904 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.822178 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.822179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.822219 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.822276 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.822360 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:23.837611 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44103
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:23.919983 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:23.920217 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:23.920246 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:23.921422 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:23.921461 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:23.921482 30020 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:22:23.921494 30020 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:23.921512 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:23.923053 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:23.923341 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:23.923552 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:23.928561 30028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.928565 30026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:23.928586 30025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:23.928807 30020 server_base.cc:1034] running on GCE node
I20260929 03:22:23.929000 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:23.929235 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:23.930410 30020 hybrid_clock.cc:648] HybridClock initialized: now 1790652143930386 us; error 33 us; skew 500 ppm
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:23.933049 30020 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:23.933506 30020 webserver.cc:460] Webserver started at http://127.27.68.67:37029/ using document root <none> and password file <none>
I20260929 03:22:23.933807 30020 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:23.933859 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:23.934018 30020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:23.935138 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "8e1906d40505465b942f349d378f3ab2"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "6d64152d5842ef0e31709b33e15015fb"
server_key_iv: "ce95ea390cdde01eed54eab6dacb6a11"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.935601 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "8e1906d40505465b942f349d378f3ab2"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "6d64152d5842ef0e31709b33e15015fb"
server_key_iv: "ce95ea390cdde01eed54eab6dacb6a11"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.936969 30020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:23.937678 30035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.937901 30020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:23.937992 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "8e1906d40505465b942f349d378f3ab2"
format_stamp: "Formatted at 2026-09-29 03:22:23 on dist-test-slave-fzwk"
server_key: "6d64152d5842ef0e31709b33e15015fb"
server_key_iv: "ce95ea390cdde01eed54eab6dacb6a11"
server_key_version: "encryptionkey@0"
I20260929 03:22:23.938092 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:23.945022 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:23.946859 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:23.946979 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:23.947554 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:23.947602 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.947628 30020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:23.947639 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:23.954912 30020 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:39517
I20260929 03:22:23.955052 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:39517 every 8 connection(s)
I20260929 03:22:23.955371 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:23.959018 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.948086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:46907 (local address 127.27.68.126:44103)
0929 03:22:23.948160 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.948163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.948437 (+ 274us) server_negotiation.cc:407] Connection header received
0929 03:22:23.949199 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.949200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.949212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.949267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.950262 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.951110 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.952963 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.953180 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.953245 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.956113 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.956115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.956115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.956124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.957615 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.958029 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.958030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.958031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.958092 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.958361 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.958362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.958362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.958480 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.958555 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.958844 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.958925 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:22:23.959254 30044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.948054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.67:46907)
0929 03:22:23.948281 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.948335 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.949075 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.949357 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.949362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.949828 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.950163 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.950172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.951267 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.951268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.952836 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.952841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.953280 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.953281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.953456 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.954147 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.954169 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.955996 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.957707 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.957714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.957717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.957948 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.958158 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.958158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.958159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.958278 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.958546 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.958552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.958706 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.958812 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.958978 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":107,"threads_started":1}
I20260929 03:22:23.959731 30151 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44103
I20260929 03:22:23.959820 30151 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:23.960021 30151 heartbeater.cc:511] Master 127.27.68.126:44103 requested a full tablet report, sending...
I20260929 03:22:23.960402 29667 ts_manager.cc:194] Registered new tserver with Master: 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:23.960774 29667 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:46907
I20260929 03:22:23.962533 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30020
I20260929 03:22:23.962635 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:23.962838 27921 external_mini_cluster.cc:1428] Setting key 474e3f077268c5241b5ab119cb7a3fd1
I20260929 03:22:23.963763 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:22:23.966885 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.960864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:40637 (local address 127.27.68.126:44103)
0929 03:22:23.960903 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.960905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.960985 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:22:23.961055 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.961056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.961066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.961127 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.961497 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.962018 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.962895 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.963160 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.963304 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.964230 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.964231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.964231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.964239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.965813 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.966206 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.966207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.966207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.966256 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.966450 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.966450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.966450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.966545 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.966594 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.966766 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.966824 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:22:23.967005 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.960799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.27.68.67:40637)
0929 03:22:23.960910 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.960920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.960997 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.961132 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.961133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.961360 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:23.961420 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.961421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.962132 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.962133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.962818 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.962820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.963304 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.963305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.963382 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.963857 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:23.963862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:23.964147 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:23.965896 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.965897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.965898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.966136 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.966315 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:23.966316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:23.966317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:23.966386 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:23.966606 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:23.966607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:23.966657 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.966740 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.966856 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:22:23 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:23.987007 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.980463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.27.68.126:44103)
0929 03:22:23.980527 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.980529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.980538 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:23.980564 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.980565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.980573 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.980639 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:23.981154 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.981642 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.982363 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.982540 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.982585 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.984937 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:23.984939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:23.984939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:23.984946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:23.986118 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.986424 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.986425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.986425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.986474 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:23.986629 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:23.986629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:23.986630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:23.986730 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:23.986801 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.986884 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.986939 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:22:23.988921 29667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:23.990723 29667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:23.997566 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.27.68.66:43575)
0929 03:22:23.993540 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.993545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.993701 (+ 156us) server_negotiation.cc:407] Connection header received
0929 03:22:23.993787 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.993790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.993806 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.993912 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:23.994094 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.995297 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.996580 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997098 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997204 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997274 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.997405 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.997444 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 03:22:23.997553 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43575 (local address 127.0.0.1:52690)
0929 03:22:23.993637 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.993647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.993696 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.993924 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.993927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.993929 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:23.994000 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.994002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.995383 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.995384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.996514 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.996516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997206 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.997288 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997303 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.997378 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.997439 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":42,"thread_start_us":68,"threads_started":1}
I20260929 03:22:23.997794 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.27.68.65:40861)
0929 03:22:23.993427 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.993432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.993623 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:22:23.993671 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.993673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.993690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.993788 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:23.994112 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.995296 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.996433 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997205 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997280 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997336 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.997614 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.997658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260929 03:22:23.997815 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:39517 (local address 127.0.0.1:52194)
0929 03:22:23.993471 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.993479 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.993527 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.994147 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.994148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.994150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:23.994203 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.994204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.995093 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.995094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.996446 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.996449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997393 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.997458 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997467 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.997518 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.997568 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260929 03:22:23.998018 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.0.0.1:52310)
0929 03:22:23.993399 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:23.993446 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:23.993527 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:23.993981 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:23.993982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:23.993983 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:23.994020 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.994022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.995399 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.995400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.996342 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:23.996345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997373 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:23.997458 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997466 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:22:23.997518 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:22:23.997567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":6,"thread_start_us":69,"threads_started":1}
I20260929 03:22:23.998298 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:23.993195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.27.68.67:39517)
0929 03:22:23.993939 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:23.993943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:23.993952 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:23.993989 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:23.993991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:23.994006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:23.994079 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:23.994285 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.994985 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.996525 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:23.997220 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:23.997280 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:23.997335 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:23.997614 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:22:23.998175 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":96,"threads_started":1}
I20260929 03:22:23.998332 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c (DEFAULT_TABLE table=test-table [id=84083ffb52864d96b7318108fec5b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:23.998723 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436bc8cc4d9c456aa6f2a686f801e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:23.998847 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c (DEFAULT_TABLE table=test-table [id=84083ffb52864d96b7318108fec5b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:23.999101 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436bc8cc4d9c456aa6f2a686f801e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:23.999572 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c (DEFAULT_TABLE table=test-table [id=84083ffb52864d96b7318108fec5b8c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:23.999888 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436bc8cc4d9c456aa6f2a686f801e86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.001807 30174 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Bootstrap starting.
I20260929 03:22:24.002667 30174 tablet_bootstrap.cc:654] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.003038 30174 log.cc:824] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:24.003077 30175 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Bootstrap starting.
I20260929 03:22:24.003718 30175 tablet_bootstrap.cc:654] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.003896 30174 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: No bootstrap required, opened a new log
I20260929 03:22:24.003952 30175 log.cc:824] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:24.003963 30174 ts_tablet_manager.cc:1397] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:22:24.004578 30175 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: No bootstrap required, opened a new log
I20260929 03:22:24.004658 30175 ts_tablet_manager.cc:1397] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:24.006114 30175 raft_consensus.cc:348] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.006253 30175 raft_consensus.cc:374] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.006134 30174 raft_consensus.cc:348] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.006273 30175 raft_consensus.cc:729] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.006299 30174 raft_consensus.cc:374] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.006340 30174 raft_consensus.cc:729] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1906d40505465b942f349d378f3ab2, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.006405 30175 consensus_queue.cc:260] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.006454 30174 consensus_queue.cc:260] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.006526 30178 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Bootstrap starting.
I20260929 03:22:24.006721 30175 ts_tablet_manager.cc:1428] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:24.006742 30174 ts_tablet_manager.cc:1428] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:22:24.006873 30151 heartbeater.cc:503] Master 127.27.68.126:44103 was elected leader, sending a full tablet report...
I20260929 03:22:24.007150 29871 heartbeater.cc:503] Master 127.27.68.126:44103 was elected leader, sending a full tablet report...
I20260929 03:22:24.007275 30178 tablet_bootstrap.cc:654] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.007609 30178 log.cc:824] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:24.008412 30178 tablet_bootstrap.cc:492] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: No bootstrap required, opened a new log
I20260929 03:22:24.008497 30178 ts_tablet_manager.cc:1397] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:22:24.010105 30178 raft_consensus.cc:348] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.010281 30178 raft_consensus.cc:374] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.010330 30178 raft_consensus.cc:729] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.010455 30178 consensus_queue.cc:260] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.010748 30178 ts_tablet_manager.cc:1428] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:22:24.010812 30011 heartbeater.cc:503] Master 127.27.68.126:44103 was elected leader, sending a full tablet report...
W20260929 03:22:24.060969 30012 tablet.cc:2369] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:22:24.153638 29872 tablet.cc:2369] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:22:24.205981 30152 tablet.cc:2369] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:24.275408 30182 raft_consensus.cc:482] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:24.275554 30182 raft_consensus.cc:504] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.276067 30182 leader_election.cc:288] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.279042 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.276236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:39517 (local address 127.27.68.66:42373)
0929 03:22:24.276286 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.276298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.276339 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.276627 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.276629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.276631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:24.276692 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.276694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.277408 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.277410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.278200 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.278202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.278771 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.278849 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.278852 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.278927 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.278965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":33}
I20260929 03:22:24.279040 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.276416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:42373 (local address 127.27.68.67:39517)
0929 03:22:24.276479 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.276480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.276487 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:22:24.276529 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.276530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.276540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.276619 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:24.276784 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.277282 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278299 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.278683 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278773 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.278782 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.278934 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.278970 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:24.279232 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.276329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:49359 (local address 127.27.68.65:40861)
0929 03:22:24.276381 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.276383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.276640 (+ 257us) server_negotiation.cc:407] Connection header received
0929 03:22:24.276736 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.276737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.276746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.276814 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:24.276979 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.277451 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278358 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.278855 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.278944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.279098 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.279139 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:22:24.279255 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.276236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.27.68.66:49359)
0929 03:22:24.276574 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.276584 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.276667 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.276830 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.276831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.276833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:24.276910 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.276912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.277539 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.277540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.278263 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.278265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.278946 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.278948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.279030 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.279033 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.279092 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.279129 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":160,"thread_start_us":100,"threads_started":1}
I20260929 03:22:24.279443 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436bc8cc4d9c456aa6f2a686f801e86c" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2" is_pre_election: true
I20260929 03:22:24.279608 30105 raft_consensus.cc:2454] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 0.
I20260929 03:22:24.279604 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436bc8cc4d9c456aa6f2a686f801e86c" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" is_pre_election: true
I20260929 03:22:24.279809 29806 raft_consensus.cc:2454] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 0.
I20260929 03:22:24.279892 29897 leader_election.cc:302] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, 8e1906d40505465b942f349d378f3ab2; no voters:
I20260929 03:22:24.280041 30182 raft_consensus.cc:2781] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:24.280112 30182 raft_consensus.cc:482] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:24.280158 30182 raft_consensus.cc:3037] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.281122 30182 raft_consensus.cc:504] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.281278 30182 leader_election.cc:288] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 election: Requested vote from peers d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.281463 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436bc8cc4d9c456aa6f2a686f801e86c" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7"
I20260929 03:22:24.281471 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436bc8cc4d9c456aa6f2a686f801e86c" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2"
I20260929 03:22:24.281549 29806 raft_consensus.cc:3037] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.281549 30105 raft_consensus.cc:3037] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.282436 30105 raft_consensus.cc:2454] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 1.
I20260929 03:22:24.282436 29806 raft_consensus.cc:2454] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 1.
I20260929 03:22:24.282629 29897 leader_election.cc:302] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, 8e1906d40505465b942f349d378f3ab2; no voters:
I20260929 03:22:24.282779 30182 raft_consensus.cc:2781] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:24.282913 30182 raft_consensus.cc:686] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 LEADER]: Becoming Leader. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Running, Role: LEADER
I20260929 03:22:24.283015 30182 consensus_queue.cc:237] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } }
I20260929 03:22:24.283885 29668 catalog_manager.cc:5495] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66). New cstate: current_term: 1 leader_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } health_report { overall_health: HEALTHY } } }
Sep 29 03:22:24 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:24.382895 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.376024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.27.68.126:44103)
0929 03:22:24.376137 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.376139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.376196 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:22:24.376266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.376267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.376275 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.376316 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:24.376796 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.377290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.377988 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.378169 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.378226 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.380481 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:24.380483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:24.380483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:24.380490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:24.381736 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:24.382134 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:24.382136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:24.382136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:24.382195 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:24.382377 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:24.382378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:24.382378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:24.382498 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:24.382560 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.382688 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.382813 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":85}
I20260929 03:22:24.382908 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.375962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44103 (local address 127.0.0.1:36930)
0929 03:22:24.376128 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.376138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.376210 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.376339 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.376340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.376662 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:24.376717 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.376719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.377368 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.377368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.377907 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.377909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.378246 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.378247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.378291 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.378818 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:24.378823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:24.380391 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:24.381817 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:24.381826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:24.381829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:24.382033 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:24.382265 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:24.382266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:24.382266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:24.382320 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:24.382553 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:24.382560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:24.382602 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.382652 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.382754 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":119,"threads_started":1}
I20260929 03:22:24.385248 29668 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:36930:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:22:24.385488 29668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:24.386994 30085 tablet_service.cc:1467] Processing CreateTablet for tablet deb1a2e77c35427a9840c4e1bb64a628 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=892736cac6404c38a995c45a019a3108]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:24.387012 29945 tablet_service.cc:1467] Processing CreateTablet for tablet deb1a2e77c35427a9840c4e1bb64a628 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=892736cac6404c38a995c45a019a3108]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:24.387029 29805 tablet_service.cc:1467] Processing CreateTablet for tablet deb1a2e77c35427a9840c4e1bb64a628 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=892736cac6404c38a995c45a019a3108]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:24.387180 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb1a2e77c35427a9840c4e1bb64a628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.387180 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb1a2e77c35427a9840c4e1bb64a628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.387214 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb1a2e77c35427a9840c4e1bb64a628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.388652 30175 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: Bootstrap starting.
I20260929 03:22:24.388715 30178 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: Bootstrap starting.
I20260929 03:22:24.389091 30178 tablet_bootstrap.cc:654] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.389106 30175 tablet_bootstrap.cc:654] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.389434 30174 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: Bootstrap starting.
I20260929 03:22:24.389760 30178 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: No bootstrap required, opened a new log
I20260929 03:22:24.389801 30178 ts_tablet_manager.cc:1397] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:24.389909 30178 raft_consensus.cc:348] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.389957 30178 raft_consensus.cc:374] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.389971 30178 raft_consensus.cc:729] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.389990 30174 tablet_bootstrap.cc:654] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.390027 30178 consensus_queue.cc:260] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.390151 30178 ts_tablet_manager.cc:1428] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:24.390226 30182 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.390332 30182 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.390588 30175 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: No bootstrap required, opened a new log
I20260929 03:22:24.390646 30175 ts_tablet_manager.cc:1397] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:24.390836 30175 raft_consensus.cc:348] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.390914 30175 raft_consensus.cc:374] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.390934 30175 raft_consensus.cc:729] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.390990 30175 consensus_queue.cc:260] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.391099 30175 ts_tablet_manager.cc:1428] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:24.391194 30179 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.391359 30179 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.391378 30174 tablet_bootstrap.cc:492] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: No bootstrap required, opened a new log
I20260929 03:22:24.391412 30174 ts_tablet_manager.cc:1397] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:22:24.391580 30174 raft_consensus.cc:348] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.391674 30174 raft_consensus.cc:374] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.391697 30174 raft_consensus.cc:729] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1906d40505465b942f349d378f3ab2, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.391746 30174 consensus_queue.cc:260] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.391855 30174 ts_tablet_manager.cc:1428] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:24.391925 30180 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.392007 30180 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.607745 30179 raft_consensus.cc:482] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:24.607856 30179 raft_consensus.cc:504] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.608271 30179 leader_election.cc:288] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.611469 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.608477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43575 (local address 127.27.68.65:45309)
0929 03:22:24.608646 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.608657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.608750 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.608944 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.608945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.608948 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:24.609025 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.609026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.609675 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.609676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.610541 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.610543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.611140 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.611141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.611261 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.611265 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.611326 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.611369 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":24,"thread_start_us":73,"threads_started":1}
I20260929 03:22:24.611454 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.608537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:45309 (local address 127.27.68.66:43575)
0929 03:22:24.608644 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.608646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.608733 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:22:24.608828 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.608829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.608839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.608940 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:24.609104 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.609586 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.610640 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.611027 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.611126 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.611139 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.611335 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.611377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260929 03:22:24.611789 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.608600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:51893 (local address 127.27.68.67:39517)
0929 03:22:24.608644 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.608646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.608715 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:22:24.608801 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.608803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.608812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.608887 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:24.609063 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.609559 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.610973 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.611418 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.611499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.611507 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.611677 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.611719 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:24.611804 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb1a2e77c35427a9840c4e1bb64a628" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" is_pre_election: true
I20260929 03:22:24.611925 29965 raft_consensus.cc:2454] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:24.612069 30196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.608472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:39517 (local address 127.27.68.65:51893)
0929 03:22:24.608631 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.608643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.608734 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.608891 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.608892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.608894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:24.608976 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.608978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.609652 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.609653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.610878 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.610881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.611499 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.611500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.611578 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.611583 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.611888 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.611931 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":56,"thread_start_us":84,"threads_started":1}
I20260929 03:22:24.612120 29758 leader_election.cc:302] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:24.612270 30179 raft_consensus.cc:2781] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:24.612306 30179 raft_consensus.cc:482] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:24.612327 30179 raft_consensus.cc:3037] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.612337 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb1a2e77c35427a9840c4e1bb64a628" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2" is_pre_election: true
I20260929 03:22:24.612418 30105 raft_consensus.cc:2454] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:24.613089 30179 raft_consensus.cc:504] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.613238 30179 leader_election.cc:288] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Requested vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.613425 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb1a2e77c35427a9840c4e1bb64a628" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2"
I20260929 03:22:24.613464 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb1a2e77c35427a9840c4e1bb64a628" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
I20260929 03:22:24.613516 30105 raft_consensus.cc:3037] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.613524 29965 raft_consensus.cc:3037] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.614166 30105 raft_consensus.cc:2454] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:24.614167 29965 raft_consensus.cc:2454] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:24.614346 29757 leader_election.cc:302] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1906d40505465b942f349d378f3ab2, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:24.614480 30179 raft_consensus.cc:2781] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:24.614614 30179 raft_consensus.cc:686] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 LEADER]: Becoming Leader. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Running, Role: LEADER
I20260929 03:22:24.614706 30179 consensus_queue.cc:237] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.615160 30198 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: TxnStatusTablet state changed. Reason: New leader d1b085ddb0bf4ac5abc23c527e22bce7. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.615298 30198 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:24.615571 30200 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:22:24.615494 29668 catalog_manager.cc:5495] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:24.615686 30200 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:22:24.676857 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.674034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.27.68.65:40861)
0929 03:22:24.674078 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.674081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.674124 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:22:24.674233 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.674233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.674249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.674306 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.674476 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.674945 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.676089 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.676280 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.676334 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.676553 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.676651 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.676712 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.676778 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.676796 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:24.676915 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.673974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.0.0.1:52318)
0929 03:22:24.674069 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.674081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.674142 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.674341 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.674342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.674343 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:24.674404 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.674407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.675037 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.675038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.675997 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.676000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.676360 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.676361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.676436 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.676457 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.676712 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.676720 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.676775 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.676828 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 03:22:24.678668 30105 raft_consensus.cc:1261] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.678659 29965 raft_consensus.cc:1261] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.678996 30198 consensus_queue.cc:1035] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:24.679169 30179 consensus_queue.cc:1035] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:24.680409 30182 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: TxnStatusTablet state changed. Reason: New leader d1b085ddb0bf4ac5abc23c527e22bce7. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.680505 30182 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.680474 30180 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: TxnStatusTablet state changed. Reason: New leader d1b085ddb0bf4ac5abc23c527e22bce7. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.680567 30180 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.681504 30198 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.681600 30198 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:24.681614 30182 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.681694 30182 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 0fedd2df28b44996a3eb5fc51bc3ce13: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.682317 30179 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.682427 30179 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P d1b085ddb0bf4ac5abc23c527e22bce7: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:24.683241 30204 mvcc.cc:205] Tried to move back new op lower bound from 7334511184601128960 to 7334511184342261760. Current Snapshot: MvccSnapshot[applied={T|T < 7334511184601128960}]
I20260929 03:22:24.683192 30180 tablet_replica.cc:439] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } } }
I20260929 03:22:24.683327 30180 tablet_replica.cc:442] T deb1a2e77c35427a9840c4e1bb64a628 P 8e1906d40505465b942f349d378f3ab2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:24.684168 30205 mvcc.cc:205] Tried to move back new op lower bound from 7334511184601128960 to 7334511184342261760. Current Snapshot: MvccSnapshot[applied={T|T < 7334511184601128960 or (T in {7334511184601128960})}]
I20260929 03:22:24.687439 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.684589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.27.68.66:43575)
0929 03:22:24.684637 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.684639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.684765 (+ 126us) server_negotiation.cc:407] Connection header received
0929 03:22:24.684829 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.684830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.684847 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.684900 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.685052 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.685517 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.686641 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.686825 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.686876 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.687065 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.687160 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.687270 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.687321 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.687347 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:22:24.692972 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.690170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:34939 (local address 127.27.68.65:40861)
0929 03:22:24.690223 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.690225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.690364 (+ 139us) server_negotiation.cc:407] Connection header received
0929 03:22:24.690427 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.690428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.690438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.690516 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.690676 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.691151 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.691936 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.692183 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.692255 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.692525 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.692595 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.692694 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.692843 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.692868 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:24.693378 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.690069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.27.68.66:34939)
0929 03:22:24.690281 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.690294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.690372 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.690521 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.690522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.690523 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:24.690596 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.690598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.691257 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.691258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.691847 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.691848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.692292 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.692293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.692365 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.692381 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.692717 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.692720 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.693184 (+ 464us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.693236 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260929 03:22:24.695854 30105 raft_consensus.cc:1261] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Refusing update from remote peer 0fedd2df28b44996a3eb5fc51bc3ce13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.695933 29806 raft_consensus.cc:1261] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Refusing update from remote peer 0fedd2df28b44996a3eb5fc51bc3ce13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.696218 30184 consensus_queue.cc:1035] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:24.696352 30182 consensus_queue.cc:1035] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:24.700314 30203 mvcc.cc:205] Tried to move back new op lower bound from 7334511184672141312 to 7334511182983786496. Current Snapshot: MvccSnapshot[applied={T|T < 7334511184672141312}]
I20260929 03:22:24.705866 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.702685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:50907 (local address 127.27.68.66:43575)
0929 03:22:24.702724 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.702727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.702884 (+ 157us) server_negotiation.cc:407] Connection header received
0929 03:22:24.702952 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.702953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.702965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.703008 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.703164 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.703685 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.705144 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.705359 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.705436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.705622 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.705656 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.705703 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.705765 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.705786 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:22:24.705904 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.702656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43575 (local address 127.27.68.65:50907)
0929 03:22:24.702814 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.702824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.702900 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.703022 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.703023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.703024 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:24.703090 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.703092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.703772 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.703773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.705054 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.705057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.705435 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.705436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.705513 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.705529 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.705708 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.705710 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.705760 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.705803 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260929 03:22:24.714499 29667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:24.714633 29667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:24.715452 29667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Sending DeleteTablet for 3 replicas of tablet 436bc8cc4d9c456aa6f2a686f801e86c
I20260929 03:22:24.715828 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:24 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52310
I20260929 03:22:24.715828 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:24 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52194
I20260929 03:22:24.716351 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 436bc8cc4d9c456aa6f2a686f801e86c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:24 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52690
I20260929 03:22:24.716372 29667 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:24.716512 29667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:24.716913 30223 tablet_replica.cc:330] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: stopping tablet replica
I20260929 03:22:24.717638 30223 raft_consensus.cc:2229] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:24.717779 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 1334c1e093294a13a212546ec1ddfe7c (DEFAULT_TABLE table=test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:24.717792 30223 raft_consensus.cc:2258] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:24.717921 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1334c1e093294a13a212546ec1ddfe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.719237 30174 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Bootstrap starting.
I20260929 03:22:24.719329 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 1334c1e093294a13a212546ec1ddfe7c (DEFAULT_TABLE table=test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:24.719439 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1334c1e093294a13a212546ec1ddfe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.719671 30174 tablet_bootstrap.cc:654] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.720352 30174 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: No bootstrap required, opened a new log
I20260929 03:22:24.720395 30174 ts_tablet_manager.cc:1397] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:24.720578 30178 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Bootstrap starting.
I20260929 03:22:24.720534 30174 raft_consensus.cc:348] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.720609 30174 raft_consensus.cc:374] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.720633 30174 raft_consensus.cc:729] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1906d40505465b942f349d378f3ab2, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.720672 30174 consensus_queue.cc:260] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.720789 30174 ts_tablet_manager.cc:1428] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:24.720995 30178 tablet_bootstrap.cc:654] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.721741 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 1334c1e093294a13a212546ec1ddfe7c (DEFAULT_TABLE table=test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:24.721861 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1334c1e093294a13a212546ec1ddfe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:24.722815 30175 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Bootstrap starting.
I20260929 03:22:24.723381 30175 tablet_bootstrap.cc:654] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:24.723850 30178 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: No bootstrap required, opened a new log
I20260929 03:22:24.723904 30178 ts_tablet_manager.cc:1397] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 03:22:24.724052 30178 raft_consensus.cc:348] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.724133 30178 raft_consensus.cc:374] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.724154 30178 raft_consensus.cc:729] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.724211 30178 consensus_queue.cc:260] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.724314 30178 ts_tablet_manager.cc:1428] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:24.725030 30223 ts_tablet_manager.cc:1905] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:24.726419 30224 tablet_replica.cc:330] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: stopping tablet replica
I20260929 03:22:24.726576 30224 raft_consensus.cc:2229] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:24.726733 30223 ts_tablet_manager.cc:1918] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:24.726827 30223 log.cc:1196] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/436bc8cc4d9c456aa6f2a686f801e86c
I20260929 03:22:24.727110 30223 ts_tablet_manager.cc:1939] T 436bc8cc4d9c456aa6f2a686f801e86c P 8e1906d40505465b942f349d378f3ab2: Deleting consensus metadata
I20260929 03:22:24.727309 30225 tablet_replica.cc:330] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: stopping tablet replica
I20260929 03:22:24.727433 30225 raft_consensus.cc:2229] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:24.727543 30225 raft_consensus.cc:2258] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:24.727571 29653 catalog_manager.cc:4841] TS 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517): tablet 436bc8cc4d9c456aa6f2a686f801e86c (table test-table [id=84083ffb52864d96b7318108fec5b8c7]) successfully deleted
I20260929 03:22:24.727618 30224 raft_consensus.cc:2258] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:24.727942 30225 ts_tablet_manager.cc:1905] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:24.727970 30224 ts_tablet_manager.cc:1905] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:24.729291 30224 ts_tablet_manager.cc:1918] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:24.729383 30224 log.cc:1196] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/436bc8cc4d9c456aa6f2a686f801e86c
I20260929 03:22:24.729636 30224 ts_tablet_manager.cc:1939] T 436bc8cc4d9c456aa6f2a686f801e86c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting consensus metadata
I20260929 03:22:24.729627 30225 ts_tablet_manager.cc:1918] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:24.729709 30225 log.cc:1196] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/436bc8cc4d9c456aa6f2a686f801e86c
I20260929 03:22:24.729964 30225 ts_tablet_manager.cc:1939] T 436bc8cc4d9c456aa6f2a686f801e86c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting consensus metadata
I20260929 03:22:24.730069 29654 catalog_manager.cc:4841] TS d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861): tablet 436bc8cc4d9c456aa6f2a686f801e86c (table test-table [id=84083ffb52864d96b7318108fec5b8c7]) successfully deleted
I20260929 03:22:24.730353 29654 catalog_manager.cc:4841] TS 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575): tablet 436bc8cc4d9c456aa6f2a686f801e86c (table test-table [id=84083ffb52864d96b7318108fec5b8c7]) successfully deleted
I20260929 03:22:24.730636 30175 tablet_bootstrap.cc:492] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: No bootstrap required, opened a new log
I20260929 03:22:24.730727 30175 ts_tablet_manager.cc:1397] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260929 03:22:24.730886 30175 raft_consensus.cc:348] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.730973 30175 raft_consensus.cc:374] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:24.731000 30175 raft_consensus.cc:729] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Initialized, Role: FOLLOWER
I20260929 03:22:24.731050 30175 consensus_queue.cc:260] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.731163 30175 ts_tablet_manager.cc:1428] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:24.770238 30179 raft_consensus.cc:482] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:24.770372 30179 raft_consensus.cc:504] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.770553 30179 leader_election.cc:288] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.770797 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1334c1e093294a13a212546ec1ddfe7c" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" is_pre_election: true
I20260929 03:22:24.770892 29965 raft_consensus.cc:2454] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:24.771009 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1334c1e093294a13a212546ec1ddfe7c" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2" is_pre_election: true
I20260929 03:22:24.771118 30105 raft_consensus.cc:2454] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:24.771314 29757 leader_election.cc:302] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1906d40505465b942f349d378f3ab2, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:24.771401 30179 raft_consensus.cc:2781] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:24.771432 30179 raft_consensus.cc:482] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:24.771459 30179 raft_consensus.cc:3037] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.772132 30179 raft_consensus.cc:504] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.772293 30179 leader_election.cc:288] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Requested vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:24.772472 30105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1334c1e093294a13a212546ec1ddfe7c" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2"
I20260929 03:22:24.772495 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1334c1e093294a13a212546ec1ddfe7c" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
I20260929 03:22:24.772541 30105 raft_consensus.cc:3037] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.772558 29965 raft_consensus.cc:3037] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:24.773324 29965 raft_consensus.cc:2454] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:24.773322 30105 raft_consensus.cc:2454] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:24.773487 29758 leader_election.cc:302] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:24.773597 30179 raft_consensus.cc:2781] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:24.773638 30179 raft_consensus.cc:686] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 LEADER]: Becoming Leader. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Running, Role: LEADER
I20260929 03:22:24.773693 30179 consensus_queue.cc:237] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:24.774240 29667 catalog_manager.cc:5495] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:24.783080 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.780826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.27.68.65:40861)
0929 03:22:24.780871 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.780873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.780882 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:24.780941 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.780943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.780955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.781009 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.781117 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.781597 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.782352 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.782561 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.782648 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.782798 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.782833 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.782905 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.782963 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.782987 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:24.787468 30220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.785272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.27.68.65:59079)
0929 03:22:24.785326 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:24.785337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:24.785400 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:24.785522 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:24.785522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:24.785523 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:24.785569 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.785571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.786142 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.786143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.786670 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:24.786672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.787049 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.787049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:24.787104 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.787109 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.787302 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.787303 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:24.787354 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:22:24.787405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":5}
I20260929 03:22:24.787468 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:24.785341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:59079 (local address 127.27.68.65:40861)
0929 03:22:24.785392 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:24.785394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:24.785402 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:24.785459 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:24.785459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:24.785468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:24.785519 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:24.785622 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.786064 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.786764 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:24.786964 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:24.787049 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:24.787185 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:24.787245 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:24.787300 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:24.787359 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:22:24.787382 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":35}
I20260929 03:22:24.789440 30105 raft_consensus.cc:1261] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.789443 29965 raft_consensus.cc:1261] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:24.789649 30179 consensus_queue.cc:1035] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:24.789798 30179 consensus_queue.cc:1035] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:22:24.794763 30203 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:22:26.314307 29668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:26.314425 29668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:26.315335 29668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Sending DeleteTablet for 3 replicas of tablet 1334c1e093294a13a212546ec1ddfe7c
I20260929 03:22:26.315670 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 1334c1e093294a13a212546ec1ddfe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52310
I20260929 03:22:26.315657 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 1334c1e093294a13a212546ec1ddfe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52194
I20260929 03:22:26.315675 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 1334c1e093294a13a212546ec1ddfe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52690
I20260929 03:22:26.315879 30253 tablet_replica.cc:330] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: stopping tablet replica
I20260929 03:22:26.315939 30253 raft_consensus.cc:2229] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:26.316002 30255 tablet_replica.cc:330] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: stopping tablet replica
I20260929 03:22:26.316068 30253 raft_consensus.cc:2258] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.316082 30255 raft_consensus.cc:2229] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:26.316123 30255 raft_consensus.cc:2258] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.316346 30253 ts_tablet_manager.cc:1905] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:26.316375 30255 ts_tablet_manager.cc:1905] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:26.317992 30255 ts_tablet_manager.cc:1918] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:22:26.318090 30255 log.cc:1196] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/1334c1e093294a13a212546ec1ddfe7c
I20260929 03:22:26.318322 30255 ts_tablet_manager.cc:1939] T 1334c1e093294a13a212546ec1ddfe7c P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting consensus metadata
I20260929 03:22:26.318488 30253 ts_tablet_manager.cc:1918] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:22:26.318567 30253 log.cc:1196] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/1334c1e093294a13a212546ec1ddfe7c
I20260929 03:22:26.318686 29654 catalog_manager.cc:4841] TS 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575): tablet 1334c1e093294a13a212546ec1ddfe7c (table test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]) successfully deleted
I20260929 03:22:26.318863 30253 ts_tablet_manager.cc:1939] T 1334c1e093294a13a212546ec1ddfe7c P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting consensus metadata
I20260929 03:22:26.319262 30254 tablet_replica.cc:330] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: stopping tablet replica
I20260929 03:22:26.319278 29654 catalog_manager.cc:4841] TS d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861): tablet 1334c1e093294a13a212546ec1ddfe7c (table test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]) successfully deleted
I20260929 03:22:26.319345 30254 raft_consensus.cc:2229] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:26.319404 30254 raft_consensus.cc:2258] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.319682 30254 ts_tablet_manager.cc:1905] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:26.321260 30254 ts_tablet_manager.cc:1918] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:22:26.321338 30254 log.cc:1196] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/1334c1e093294a13a212546ec1ddfe7c
I20260929 03:22:26.321537 30254 ts_tablet_manager.cc:1939] T 1334c1e093294a13a212546ec1ddfe7c P 8e1906d40505465b942f349d378f3ab2: Deleting consensus metadata
I20260929 03:22:26.321877 29653 catalog_manager.cc:4841] TS 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517): tablet 1334c1e093294a13a212546ec1ddfe7c (table test-table [id=78a4b3bb67974e46bc9c0e9885ecb20e]) successfully deleted
Sep 29 03:22:26 dist-test-slave-fzwk krb5kdc[29623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652146, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:22:26 dist-test-slave-fzwk krb5kdc[29623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652146, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:26.351166 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:26.334843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.27.68.126:44103)
0929 03:22:26.334997 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:26.334999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:26.335013 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:22:26.335097 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:26.335098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:26.335109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:26.335178 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:26.335864 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.336384 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.337230 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.337426 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.337480 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:26.348478 (+ 10998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:26.348480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:26.348480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:26.348489 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:26.350126 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:26.350488 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:26.350489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:26.350489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:26.350546 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:26.350715 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:26.350716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:26.350716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:26.350831 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:26.350908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:26.351002 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:22:26.351078 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260929 03:22:26.352699 29668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:26.352885 29668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:26.354271 29805 tablet_service.cc:1467] Processing CreateTablet for tablet cbefcce5baae498fb095c5725555d124 (DEFAULT_TABLE table=test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.354278 29945 tablet_service.cc:1467] Processing CreateTablet for tablet cbefcce5baae498fb095c5725555d124 (DEFAULT_TABLE table=test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.354321 30085 tablet_service.cc:1467] Processing CreateTablet for tablet cbefcce5baae498fb095c5725555d124 (DEFAULT_TABLE table=test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.354429 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbefcce5baae498fb095c5725555d124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.354429 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbefcce5baae498fb095c5725555d124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.354439 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbefcce5baae498fb095c5725555d124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.355801 30264 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Bootstrap starting.
I20260929 03:22:26.355984 30266 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Bootstrap starting.
I20260929 03:22:26.355985 30265 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Bootstrap starting.
I20260929 03:22:26.356334 30264 tablet_bootstrap.cc:654] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.356382 30266 tablet_bootstrap.cc:654] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.356382 30265 tablet_bootstrap.cc:654] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.356994 30264 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: No bootstrap required, opened a new log
I20260929 03:22:26.357057 30264 ts_tablet_manager.cc:1397] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:26.357254 30265 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: No bootstrap required, opened a new log
I20260929 03:22:26.357308 30265 ts_tablet_manager.cc:1397] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:26.357369 30266 tablet_bootstrap.cc:492] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: No bootstrap required, opened a new log
I20260929 03:22:26.357429 30266 ts_tablet_manager.cc:1397] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:26.357455 30265 raft_consensus.cc:348] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357213 30264 raft_consensus.cc:348] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357524 30265 raft_consensus.cc:374] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.357547 30264 raft_consensus.cc:374] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.357579 30264 raft_consensus.cc:729] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.357589 30265 raft_consensus.cc:729] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1906d40505465b942f349d378f3ab2, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.357570 30266 raft_consensus.cc:348] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357646 30266 raft_consensus.cc:374] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.357671 30266 raft_consensus.cc:729] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.357648 30264 consensus_queue.cc:260] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357661 30265 consensus_queue.cc:260] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357733 30266 consensus_queue.cc:260] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.357821 30264 ts_tablet_manager.cc:1428] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:26.357851 30266 ts_tablet_manager.cc:1428] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:26.357961 30265 ts_tablet_manager.cc:1428] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:26.361805 30270 raft_consensus.cc:482] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:26.361883 30270 raft_consensus.cc:504] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.362020 30270 leader_election.cc:288] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:26.362205 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbefcce5baae498fb095c5725555d124" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" is_pre_election: true
I20260929 03:22:26.362315 29806 raft_consensus.cc:2454] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 0.
I20260929 03:22:26.362285 30104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbefcce5baae498fb095c5725555d124" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2" is_pre_election: true
I20260929 03:22:26.362387 30104 raft_consensus.cc:2454] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 0.
I20260929 03:22:26.362573 29898 leader_election.cc:302] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:26.362664 30270 raft_consensus.cc:2781] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:26.362715 30270 raft_consensus.cc:482] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:26.362737 30270 raft_consensus.cc:3037] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.363493 30270 raft_consensus.cc:504] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.363626 30270 leader_election.cc:288] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 election: Requested vote from peers d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:26.363760 30104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbefcce5baae498fb095c5725555d124" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2"
I20260929 03:22:26.363750 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbefcce5baae498fb095c5725555d124" candidate_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7"
I20260929 03:22:26.363817 30104 raft_consensus.cc:3037] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.363817 29806 raft_consensus.cc:3037] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.364559 29806 raft_consensus.cc:2454] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 1.
I20260929 03:22:26.364559 30104 raft_consensus.cc:2454] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fedd2df28b44996a3eb5fc51bc3ce13 in term 1.
I20260929 03:22:26.364714 29897 leader_election.cc:302] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, 8e1906d40505465b942f349d378f3ab2; no voters:
I20260929 03:22:26.364820 30270 raft_consensus.cc:2781] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:26.364940 30270 raft_consensus.cc:686] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 LEADER]: Becoming Leader. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Running, Role: LEADER
I20260929 03:22:26.364980 30270 consensus_queue.cc:237] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.365511 29668 catalog_manager.cc:5495] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66). New cstate: current_term: 1 leader_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:26.372865 30275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:26.369639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.27.68.66:43575)
0929 03:22:26.369769 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:26.369771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:26.369778 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:22:26.369804 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:26.369804 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:26.369813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:26.369886 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:26.370039 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.370729 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.371907 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.372279 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.372385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:26.372607 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:26.372646 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:26.372702 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:26.372753 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:22:26.372778 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260929 03:22:26.375554 29806 raft_consensus.cc:1261] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Refusing update from remote peer 0fedd2df28b44996a3eb5fc51bc3ce13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:26.375566 30104 raft_consensus.cc:1261] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Refusing update from remote peer 0fedd2df28b44996a3eb5fc51bc3ce13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:26.375773 30270 consensus_queue.cc:1035] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:26.375872 30272 consensus_queue.cc:1035] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:26.388922 29666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:26.389029 29666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:26.389668 29666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Sending DeleteTablet for 3 replicas of tablet cbefcce5baae498fb095c5725555d124
I20260929 03:22:26.389921 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet cbefcce5baae498fb095c5725555d124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52690
I20260929 03:22:26.389932 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet cbefcce5baae498fb095c5725555d124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52194
I20260929 03:22:26.389940 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet cbefcce5baae498fb095c5725555d124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:26 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52310
I20260929 03:22:26.390044 30253 tablet_replica.cc:330] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: stopping tablet replica
I20260929 03:22:26.390048 30255 tablet_replica.cc:330] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: stopping tablet replica
I20260929 03:22:26.390054 30254 tablet_replica.cc:330] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: stopping tablet replica
I20260929 03:22:26.390101 30253 raft_consensus.cc:2229] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:26.390105 30254 raft_consensus.cc:2229] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:26.390106 30255 raft_consensus.cc:2229] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:26.390138 30253 raft_consensus.cc:2258] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.390142 30254 raft_consensus.cc:2258] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.390173 30255 raft_consensus.cc:2258] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:26.390441 30255 ts_tablet_manager.cc:1905] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:26.390437 29666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:22:26.390568 30254 ts_tablet_manager.cc:1905] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:22:26.390594 29666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:26.391988 30085 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 (DEFAULT_TABLE table=test-table [id=6e5d391dea3a4f5e973984cc33207542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.392117 30085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed193dcab34ce58bad551d99f6dbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.392331 29805 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 (DEFAULT_TABLE table=test-table [id=6e5d391dea3a4f5e973984cc33207542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.392362 30254 ts_tablet_manager.cc:1918] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:26.392407 30254 log.cc:1196] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/cbefcce5baae498fb095c5725555d124
I20260929 03:22:26.392446 29805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed193dcab34ce58bad551d99f6dbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.392409 30255 ts_tablet_manager.cc:1918] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:26.392495 30255 log.cc:1196] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/cbefcce5baae498fb095c5725555d124
I20260929 03:22:26.392650 30254 ts_tablet_manager.cc:1939] T cbefcce5baae498fb095c5725555d124 P 8e1906d40505465b942f349d378f3ab2: Deleting consensus metadata
I20260929 03:22:26.392764 30255 ts_tablet_manager.cc:1939] T cbefcce5baae498fb095c5725555d124 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting consensus metadata
I20260929 03:22:26.392961 29653 catalog_manager.cc:4841] TS 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517): tablet cbefcce5baae498fb095c5725555d124 (table test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]) successfully deleted
I20260929 03:22:26.393112 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 (DEFAULT_TABLE table=test-table [id=6e5d391dea3a4f5e973984cc33207542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:26.393208 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15ed193dcab34ce58bad551d99f6dbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:26.393882 30266 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7: Bootstrap starting.
I20260929 03:22:26.393941 30265 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Bootstrap starting.
I20260929 03:22:26.394037 30253 ts_tablet_manager.cc:1905] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:26.394119 30264 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Bootstrap starting.
I20260929 03:22:26.394416 30265 tablet_bootstrap.cc:654] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.394439 30264 tablet_bootstrap.cc:654] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.394459 30266 tablet_bootstrap.cc:654] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:26.394779 29654 catalog_manager.cc:4841] TS 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575): tablet cbefcce5baae498fb095c5725555d124 (table test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]) successfully deleted
I20260929 03:22:26.395358 30266 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7: No bootstrap required, opened a new log
I20260929 03:22:26.395427 30266 ts_tablet_manager.cc:1397] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:26.395601 30253 ts_tablet_manager.cc:1918] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:26.395591 30266 raft_consensus.cc:348] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.395690 30253 log.cc:1196] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/cbefcce5baae498fb095c5725555d124
I20260929 03:22:26.395689 30266 raft_consensus.cc:374] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.395767 30266 raft_consensus.cc:729] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.395843 30266 consensus_queue.cc:260] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.395946 30265 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: No bootstrap required, opened a new log
I20260929 03:22:26.395953 30264 tablet_bootstrap.cc:492] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: No bootstrap required, opened a new log
I20260929 03:22:26.395956 30253 ts_tablet_manager.cc:1939] T cbefcce5baae498fb095c5725555d124 P d1b085ddb0bf4ac5abc23c527e22bce7: Deleting consensus metadata
I20260929 03:22:26.395995 30264 ts_tablet_manager.cc:1397] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:26.395989 30265 ts_tablet_manager.cc:1397] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:26.396104 30264 raft_consensus.cc:348] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.396167 30264 raft_consensus.cc:374] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.396173 30266 ts_tablet_manager.cc:1428] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:26.396196 30264 raft_consensus.cc:729] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fedd2df28b44996a3eb5fc51bc3ce13, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.396150 30265 raft_consensus.cc:348] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.396219 30265 raft_consensus.cc:374] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:26.396241 30265 raft_consensus.cc:729] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1906d40505465b942f349d378f3ab2, State: Initialized, Role: FOLLOWER
I20260929 03:22:26.396238 30264 consensus_queue.cc:260] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.396318 30264 ts_tablet_manager.cc:1428] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:26.396289 30265 consensus_queue.cc:260] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.396395 30265 ts_tablet_manager.cc:1428] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:26.396664 29654 catalog_manager.cc:4841] TS d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861): tablet cbefcce5baae498fb095c5725555d124 (table test-table [id=ad1d45ac5b584c01a2c35583b559dfa9]) successfully deleted
I20260929 03:22:26.547593 30198 raft_consensus.cc:482] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:26.547721 30198 raft_consensus.cc:504] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.547948 30198 leader_election.cc:288] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:26.548162 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ed193dcab34ce58bad551d99f6dbf1" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" is_pre_election: true
I20260929 03:22:26.548228 30104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ed193dcab34ce58bad551d99f6dbf1" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2" is_pre_election: true
I20260929 03:22:26.548301 29965 raft_consensus.cc:2454] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:26.548317 30104 raft_consensus.cc:2454] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 0.
I20260929 03:22:26.548502 29758 leader_election.cc:302] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:26.548643 30198 raft_consensus.cc:2781] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:26.548702 30198 raft_consensus.cc:482] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:26.548717 30198 raft_consensus.cc:3037] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.549485 30198 raft_consensus.cc:504] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.549640 30198 leader_election.cc:288] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Requested vote from peers 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575), 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517)
I20260929 03:22:26.549813 30104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ed193dcab34ce58bad551d99f6dbf1" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1906d40505465b942f349d378f3ab2"
I20260929 03:22:26.549852 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15ed193dcab34ce58bad551d99f6dbf1" candidate_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13"
I20260929 03:22:26.549908 29965 raft_consensus.cc:3037] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.549911 30104 raft_consensus.cc:3037] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:26.550508 29965 raft_consensus.cc:2454] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:26.550619 30104 raft_consensus.cc:2454] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1b085ddb0bf4ac5abc23c527e22bce7 in term 1.
I20260929 03:22:26.550691 29758 leader_election.cc:302] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fedd2df28b44996a3eb5fc51bc3ce13, d1b085ddb0bf4ac5abc23c527e22bce7; no voters:
I20260929 03:22:26.550774 30198 raft_consensus.cc:2781] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:26.550818 30198 raft_consensus.cc:686] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [term 1 LEADER]: Becoming Leader. State: Replica: d1b085ddb0bf4ac5abc23c527e22bce7, State: Running, Role: LEADER
I20260929 03:22:26.550865 30198 consensus_queue.cc:237] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } }
I20260929 03:22:26.551501 29667 catalog_manager.cc:5495] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1b085ddb0bf4ac5abc23c527e22bce7" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:26.580688 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:26.578414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.27.68.65:40861)
0929 03:22:26.578576 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:26.578578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:26.578583 (+ 5us) server_negotiation.cc:407] Connection header received
0929 03:22:26.578601 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:26.578601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:26.578611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:26.578664 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:26.578795 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.579373 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.580121 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:26.580319 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:26.580384 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:26.580486 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:26.580514 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:26.580548 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:26.580605 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:22:26.580621 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":111,"threads_started":1}
I20260929 03:22:26.582986 30104 raft_consensus.cc:1261] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:26.583005 29965 raft_consensus.cc:1261] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Refusing update from remote peer d1b085ddb0bf4ac5abc23c527e22bce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:26.583261 30248 consensus_queue.cc:1035] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1906d40505465b942f349d378f3ab2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:26.583400 30235 consensus_queue.cc:1035] T 15ed193dcab34ce58bad551d99f6dbf1 P d1b085ddb0bf4ac5abc23c527e22bce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fedd2df28b44996a3eb5fc51bc3ce13" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:22:26.587539 30233 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:22:28.104377 29668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:28.104478 29668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:28.105283 29668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6a8354a35af4fb5acffea4367e5ec17: Sending DeleteTablet for 3 replicas of tablet 15ed193dcab34ce58bad551d99f6dbf1
I20260929 03:22:28.105543 29805 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:28 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52310
I20260929 03:22:28.105568 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:28 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52690
I20260929 03:22:28.105603 30085 tablet_service.cc:1514] Processing DeleteTablet for tablet 15ed193dcab34ce58bad551d99f6dbf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:28 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:52194
I20260929 03:22:28.105672 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29740
I20260929 03:22:28.106410 30309 tablet_replica.cc:330] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: stopping tablet replica
I20260929 03:22:28.106493 30309 raft_consensus.cc:2229] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:28.106537 30309 raft_consensus.cc:2258] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:28.106853 30309 ts_tablet_manager.cc:1905] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:28.109028 30309 ts_tablet_manager.cc:1918] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:22:28.109102 30309 log.cc:1196] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/15ed193dcab34ce58bad551d99f6dbf1
I20260929 03:22:28.109328 30309 ts_tablet_manager.cc:1939] T 15ed193dcab34ce58bad551d99f6dbf1 P 0fedd2df28b44996a3eb5fc51bc3ce13: Deleting consensus metadata
I20260929 03:22:28.109509 30310 tablet_replica.cc:330] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: stopping tablet replica
I20260929 03:22:28.109586 30310 raft_consensus.cc:2229] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:28.109639 30310 raft_consensus.cc:2258] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:28.109802 29654 catalog_manager.cc:4841] TS 0fedd2df28b44996a3eb5fc51bc3ce13 (127.27.68.66:43575): tablet 15ed193dcab34ce58bad551d99f6dbf1 (table test-table [id=6e5d391dea3a4f5e973984cc33207542]) successfully deleted
I20260929 03:22:28.110026 30310 ts_tablet_manager.cc:1905] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:28.112008 30310 ts_tablet_manager.cc:1918] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:22:28.112082 30310 log.cc:1196] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/15ed193dcab34ce58bad551d99f6dbf1
I20260929 03:22:28.112325 30310 ts_tablet_manager.cc:1939] T 15ed193dcab34ce58bad551d99f6dbf1 P 8e1906d40505465b942f349d378f3ab2: Deleting consensus metadata
I20260929 03:22:28.112751 29653 catalog_manager.cc:4841] TS 8e1906d40505465b942f349d378f3ab2 (127.27.68.67:39517): tablet 15ed193dcab34ce58bad551d99f6dbf1 (table test-table [id=6e5d391dea3a4f5e973984cc33207542]) successfully deleted
W20260929 03:22:28.115072 29654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:40861 (error 108)
I20260929 03:22:28.115402 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29880
I20260929 03:22:28.116107 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:28.115577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40861 (local address 127.0.0.1:52328)
0929 03:22:28.116001 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:28.116052 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":66,"threads_started":1}
W20260929 03:22:28.116200 29654 catalog_manager.cc:4584] TS d1b085ddb0bf4ac5abc23c527e22bce7 (127.27.68.65:40861): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15ed193dcab34ce58bad551d99f6dbf1: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40861: connect: Connection refused (error 111)
I20260929 03:22:28.122066 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30020
I20260929 03:22:28.126845 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 29636
2026-09-29T03:22:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5937 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:28 dist-test-slave-fzwk krb5kdc[30315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:28 dist-test-slave-fzwk krb5kdc[30315](info): set up 2 sockets
Sep 29 03:22:28 dist-test-slave-fzwk krb5kdc[30315](info): commencing operation
krb5kdc: starting...
W20260929 03:22:29.159322 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:29.277838 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33971
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:33971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:29.360222 30328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:29.360459 30328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:29.360486 30328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:29.361783 30328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:29.361833 30328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:29.361848 30328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:29.361861 30328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:29.361873 30328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:29.363576 30328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:33971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:29.363864 30328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:29.364115 30328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:29.369654 30336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:29.369671 30328 server_base.cc:1034] running on GCE node
W20260929 03:22:29.369654 30333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.369664 30334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:29.370286 30328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:29.370656 30328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:29.371847 30328 hybrid_clock.cc:648] HybridClock initialized: now 1790652149371828 us; error 38 us; skew 500 ppm
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:29.374586 30328 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:29.375042 30328 webserver.cc:460] Webserver started at http://127.27.68.126:41921/ using document root <none> and password file <none>
I20260929 03:22:29.375334 30328 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:29.375392 30328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:29.375514 30328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:29.376631 30328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "26a0bdbeb89346359e92c5b9d88c78a7"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "8a1651c6b1c4564228512b47d3f98e80"
server_key_iv: "5cc047308190c24e0367c4f2b5946a2a"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.376982 30328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "26a0bdbeb89346359e92c5b9d88c78a7"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "8a1651c6b1c4564228512b47d3f98e80"
server_key_iv: "5cc047308190c24e0367c4f2b5946a2a"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.378444 30328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:29.379310 30343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.379524 30328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:29.379633 30328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "26a0bdbeb89346359e92c5b9d88c78a7"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "8a1651c6b1c4564228512b47d3f98e80"
server_key_iv: "5cc047308190c24e0367c4f2b5946a2a"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.379712 30328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:29.388109 30328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:29.389896 30328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:29.390005 30328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:29.393824 30328 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:33971
I20260929 03:22:29.393878 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:33971 every 8 connection(s)
I20260929 03:22:29.394239 30328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:29.395161 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:29.397527 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: Bootstrap starting.
I20260929 03:22:29.398156 30396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:29.398449 30396 log.cc:824] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:29.399427 30396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: No bootstrap required, opened a new log
I20260929 03:22:29.401023 30396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } }
I20260929 03:22:29.401181 30396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:29.401208 30396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a0bdbeb89346359e92c5b9d88c78a7, State: Initialized, Role: FOLLOWER
I20260929 03:22:29.401387 30396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } }
I20260929 03:22:29.401454 30396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:29.401477 30396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:29.401547 30396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:29.402107 30396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } }
I20260929 03:22:29.402225 30396 leader_election.cc:302] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a0bdbeb89346359e92c5b9d88c78a7; no voters:
I20260929 03:22:29.402392 30396 leader_election.cc:288] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:29.402458 30399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:29.402660 30399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [term 1 LEADER]: Becoming Leader. State: Replica: 26a0bdbeb89346359e92c5b9d88c78a7, State: Running, Role: LEADER
I20260929 03:22:29.402685 30396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:29.402848 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30328
I20260929 03:22:29.402853 30399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } }
I20260929 03:22:29.402930 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:29.403123 27921 external_mini_cluster.cc:1428] Setting key a03c7bec9bee7c68027b016df9d3a4aa
I20260929 03:22:29.404031 30401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a0bdbeb89346359e92c5b9d88c78a7. Latest consensus state: current_term: 1 leader_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } } }
I20260929 03:22:29.404161 30401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [sys.catalog]: This master's current role is: LEADER
W20260929 03:22:29.405037 30416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 03:22:29.404476 30400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a0bdbeb89346359e92c5b9d88c78a7" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33971 } } }
W20260929 03:22:29.405385 30416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:22:29.405404 30400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:29.405529 30417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:29.405715 30417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:29.408003 30417 catalog_manager.cc:1347] Generated new cluster ID: 5158cb71033e403dbd82389fd4b0a307
I20260929 03:22:29.408067 30417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:29.414762 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.403811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.27.68.126:33971)
0929 03:22:29.404001 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:29.404012 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:29.404136 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:22:29.404307 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:29.404314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:29.404383 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:29.404533 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:29.405290 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.406482 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.407637 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.408148 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.408260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.410666 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:29.410671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:29.410671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:29.410685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:29.413066 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.413476 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.413480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.413480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.413594 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.413816 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.413817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.413818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.414099 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:29.414279 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:29.414379 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:22:29.414538 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260929 03:22:29.427536 30417 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:29.428246 30417 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:29.434836 30417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a0bdbeb89346359e92c5b9d88c78a7: Generated new TSK 0
I20260929 03:22:29.435067 30417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:29.464378 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:29.568369 30422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:29.568641 30422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:29.568686 30422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:29.569974 30422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:29.570060 30422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:29.570109 30422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:29.571768 30422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:29.572065 30422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:29.572333 30422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:29.577399 30427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.577422 30428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.577416 30430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:29.577708 30422 server_base.cc:1034] running on GCE node
I20260929 03:22:29.577940 30422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:29.578208 30422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:29.579380 30422 hybrid_clock.cc:648] HybridClock initialized: now 1790652149579362 us; error 30 us; skew 500 ppm
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:29.581959 30422 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:29.582428 30422 webserver.cc:460] Webserver started at http://127.27.68.65:46659/ using document root <none> and password file <none>
I20260929 03:22:29.582685 30422 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:29.582767 30422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:29.582923 30422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:29.584090 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "9d34652252334c25bc909e0d16368ece"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "e286999e66ddac118e55d07044cf5b08"
server_key_iv: "6744bd96b8fc1f7a58da7014e67a1098"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.584460 30422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "9d34652252334c25bc909e0d16368ece"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "e286999e66ddac118e55d07044cf5b08"
server_key_iv: "6744bd96b8fc1f7a58da7014e67a1098"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.585922 30422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:22:29.586694 30437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.586872 30422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:29.586966 30422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "9d34652252334c25bc909e0d16368ece"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "e286999e66ddac118e55d07044cf5b08"
server_key_iv: "6744bd96b8fc1f7a58da7014e67a1098"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.587049 30422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:29.607873 30422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:29.609826 30422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:29.610004 30422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:29.610268 30422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:29.610603 30422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:29.610658 30422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.610709 30422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:29.610759 30422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.616649 30422 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:42811
I20260929 03:22:29.616734 30550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:42811 every 8 connection(s)
I20260929 03:22:29.617082 30422 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:29.620314 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30422
I20260929 03:22:29.620416 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:29.620603 27921 external_mini_cluster.cc:1428] Setting key c8acb3b44cf7863ba47ffa5a6ee57122
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:29.627357 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.617868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:36829 (local address 127.27.68.126:33971)
0929 03:22:29.617937 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:29.617939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:29.618177 (+ 238us) server_negotiation.cc:407] Connection header received
0929 03:22:29.618646 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:29.618647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:29.618656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:29.618731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:29.619730 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.620208 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.621610 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.621820 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.621876 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.624292 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:29.624293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:29.624294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:29.624300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:29.625728 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.626270 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.626271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.626271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.626323 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.626651 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.626651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.626651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.626755 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:29.626806 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:29.627163 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 03:22:29.627290 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:22:29.627620 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.617848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33971 (local address 127.27.68.65:36829)
0929 03:22:29.618029 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:29.618075 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:29.618562 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:29.618785 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:29.618788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:29.619440 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:29.619641 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.619646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.620308 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.620308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.621531 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.621534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.621906 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.621908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.622112 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.622650 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:29.622662 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:29.624196 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:29.625817 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.625826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.625829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.626191 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.626390 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.626391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.626392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.626579 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.626823 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:29.626830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:29.627018 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 03:22:29.627146 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:22:29.627365 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260929 03:22:29.628248 30551 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33971
I20260929 03:22:29.628356 30551 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:29.628587 30551 heartbeater.cc:511] Master 127.27.68.126:33971 requested a full tablet report, sending...
I20260929 03:22:29.629254 30360 ts_manager.cc:194] Registered new tserver with Master: 9d34652252334c25bc909e0d16368ece (127.27.68.65:42811)
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:22:29.630129 30360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:36829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:29.646377 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:29.729640 30556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:29.729852 30556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:29.729877 30556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:29.731089 30556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:29.731132 30556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:29.731158 30556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:29.732609 30556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:29.732806 30556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:29.732985 30556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:29.737840 30562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.737846 30564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.737882 30561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:29.738114 30556 server_base.cc:1034] running on GCE node
I20260929 03:22:29.738334 30556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:29.738579 30556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:29.739727 30556 hybrid_clock.cc:648] HybridClock initialized: now 1790652149739718 us; error 28 us; skew 500 ppm
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:29.742205 30556 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:29.742710 30556 webserver.cc:460] Webserver started at http://127.27.68.66:35107/ using document root <none> and password file <none>
I20260929 03:22:29.742941 30556 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:29.742986 30556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:29.743165 30556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:29.744269 30556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "f7fc353c44a74e42a87c0302a7051265"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "c8567b4ab3133f8fa1e03cf2bf7701e4"
server_key_iv: "7f627a8833cfc6704de4c7020e00b57d"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.744626 30556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "f7fc353c44a74e42a87c0302a7051265"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "c8567b4ab3133f8fa1e03cf2bf7701e4"
server_key_iv: "7f627a8833cfc6704de4c7020e00b57d"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.746025 30556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:29.746784 30571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.746959 30556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:29.747067 30556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "f7fc353c44a74e42a87c0302a7051265"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "c8567b4ab3133f8fa1e03cf2bf7701e4"
server_key_iv: "7f627a8833cfc6704de4c7020e00b57d"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.747155 30556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:29.775830 30556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:29.777714 30556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:29.777886 30556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:29.778131 30556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:29.778456 30556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:29.778512 30556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.778549 30556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:29.778580 30556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.784897 30556 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:33395
I20260929 03:22:29.784942 30684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:33395 every 8 connection(s)
I20260929 03:22:29.785293 30556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:29.791975 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30556
I20260929 03:22:29.792088 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:29.792338 27921 external_mini_cluster.cc:1428] Setting key e27c5160993915a58bca16d8955d2bce
I20260929 03:22:29.795027 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.786133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:39701 (local address 127.27.68.126:33971)
0929 03:22:29.786208 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:29.786210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:29.786428 (+ 218us) server_negotiation.cc:407] Connection header received
0929 03:22:29.786895 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:29.786896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:29.786906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:29.786967 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:29.787992 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.788510 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.789473 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.789661 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.789719 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.791869 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:29.791871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:29.791871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:29.791880 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:29.793639 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.794022 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.794023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.794023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.794091 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.794326 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.794328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.794328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.794474 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:29.794552 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:29.794892 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 03:22:29.794962 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:22:29.795359 30687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.786124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33971 (local address 127.27.68.66:39701)
0929 03:22:29.786289 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:29.786332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:29.786811 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:29.787029 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:29.787033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:29.787685 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:29.787901 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.787906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.788595 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.788596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.789394 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.789396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.789778 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.789779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.789909 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.790374 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:29.790382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:29.791767 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:29.793718 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.793725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.793727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.793947 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.794155 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.794156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.794156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.794264 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.794563 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:29.794570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:29.794746 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 03:22:29.794870 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:22:29.795055 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260929 03:22:29.795961 30685 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33971
I20260929 03:22:29.796057 30685 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:29.796252 30685 heartbeater.cc:511] Master 127.27.68.126:33971 requested a full tablet report, sending...
I20260929 03:22:29.796639 30360 ts_manager.cc:194] Registered new tserver with Master: f7fc353c44a74e42a87c0302a7051265 (127.27.68.66:33395)
I20260929 03:22:29.797029 30360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:39701
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:29.818069 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:29.901242 30690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:29.901459 30690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:29.901484 30690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:29.902902 30690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:29.902947 30690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:29.902995 30690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:29.904979 30690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:29.905258 30690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:29.905500 30690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:29.910728 30695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.910708 30696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:29.910704 30698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:29.910745 30690 server_base.cc:1034] running on GCE node
I20260929 03:22:29.911142 30690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:29.911445 30690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:29.912607 30690 hybrid_clock.cc:648] HybridClock initialized: now 1790652149912602 us; error 39 us; skew 500 ppm
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:29.915172 30690 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:29.915652 30690 webserver.cc:460] Webserver started at http://127.27.68.67:33073/ using document root <none> and password file <none>
I20260929 03:22:29.915902 30690 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:29.915975 30690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:29.916144 30690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:29.917238 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "1e093c116ee64884b2951f5a59cc51ed"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "ab03d2569c3d2d581a65bad2843af8cc"
server_key_iv: "284ce0e245d7cb2408a94ddfd62fe23b"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.917613 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "1e093c116ee64884b2951f5a59cc51ed"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "ab03d2569c3d2d581a65bad2843af8cc"
server_key_iv: "284ce0e245d7cb2408a94ddfd62fe23b"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.918977 30690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:29.919790 30705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.919953 30690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.920050 30690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "1e093c116ee64884b2951f5a59cc51ed"
format_stamp: "Formatted at 2026-09-29 03:22:29 on dist-test-slave-fzwk"
server_key: "ab03d2569c3d2d581a65bad2843af8cc"
server_key_iv: "284ce0e245d7cb2408a94ddfd62fe23b"
server_key_version: "encryptionkey@0"
I20260929 03:22:29.920135 30690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:29.938019 30690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:29.939970 30690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:29.940137 30690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:29.940387 30690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:29.940696 30690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:29.940752 30690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.940797 30690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:29.940822 30690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:29.946801 30690 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:41635
I20260929 03:22:29.946846 30818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:41635 every 8 connection(s)
I20260929 03:22:29.947192 30690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:29.953001 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30690
I20260929 03:22:29.953115 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:29.953291 27921 external_mini_cluster.cc:1428] Setting key 8129f87cb6170772304f90f8ae10d2e6
Sep 29 03:22:29 dist-test-slave-fzwk krb5kdc[30315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652149, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:29.956652 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.947990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:46157 (local address 127.27.68.126:33971)
0929 03:22:29.948076 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:29.948078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:29.948279 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:22:29.948767 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:29.948768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:29.948780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:29.948834 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:29.949843 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.950350 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.951481 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.951671 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.951729 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.954228 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:29.954229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:29.954230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:29.954238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:29.955498 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.955872 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.955872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.955873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.955920 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:29.956139 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:29.956140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:29.956140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:29.956240 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:29.956289 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:29.956525 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 03:22:29.956587 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:22:29.956851 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:29.947964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33971 (local address 127.27.68.67:46157)
0929 03:22:29.948150 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:29.948191 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:29.948682 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:29.948893 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:29.948897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:29.949489 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:29.949719 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.949724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.950434 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.950435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.951404 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:29.951407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:29.951748 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:29.951749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:29.951856 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:29.952377 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:29.952386 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:29.954088 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:29.955570 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.955576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.955578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.955804 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.955977 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:29.955978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:29.955978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:29.956086 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:29.956293 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:29.956297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:29.956414 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 03:22:29.956497 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:29.956624 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260929 03:22:29.957343 30819 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33971
I20260929 03:22:29.957480 30819 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:29.957676 30819 heartbeater.cc:511] Master 127.27.68.126:33971 requested a full tablet report, sending...
I20260929 03:22:29.958113 30360 ts_manager.cc:194] Registered new tserver with Master: 1e093c116ee64884b2951f5a59cc51ed (127.27.68.67:41635)
I20260929 03:22:29.958500 30360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:46157
I20260929 03:22:29.964959 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:22:29.976383 30410 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:29.975365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.27.68.126:33971)
0929 03:22:29.975458 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:29.975461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:29.975518 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:22:29.975572 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:29.975573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:29.975583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:29.975636 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:29.976298 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52392: BlockingRecv error: recv got EOF from 127.0.0.1:52392 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 03:22:29.977238 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30422
I20260929 03:22:29.981781 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30556
I20260929 03:22:29.986274 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30690
I20260929 03:22:29.990696 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30328
2026-09-29T03:22:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1862 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:30 dist-test-slave-fzwk krb5kdc[30831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:30 dist-test-slave-fzwk krb5kdc[30831](info): set up 2 sockets
Sep 29 03:22:30 dist-test-slave-fzwk krb5kdc[30831](info): commencing operation
krb5kdc: starting...
W20260929 03:22:31.015518 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:31.133807 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46809
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:31.214738 30844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:31.214993 30844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:31.215034 30844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:31.216233 30844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:31.216306 30844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:31.216331 30844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:31.216358 30844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:31.216389 30844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:31.217941 30844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:31.218255 30844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:31.218490 30844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:31.223347 30849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:31.223349 30852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:31.223570 30844 server_base.cc:1034] running on GCE node
W20260929 03:22:31.223368 30850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:31.223892 30844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:31.224179 30844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:31.225353 30844 hybrid_clock.cc:648] HybridClock initialized: now 1790652151225349 us; error 35 us; skew 500 ppm
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652151, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:31.227856 30844 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:31.228370 30844 webserver.cc:460] Webserver started at http://127.27.68.126:46345/ using document root <none> and password file <none>
I20260929 03:22:31.228603 30844 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:31.228679 30844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:31.228834 30844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:31.230053 30844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "3247be068829485e9aa324625b7bf033"
format_stamp: "Formatted at 2026-09-29 03:22:31 on dist-test-slave-fzwk"
server_key: "0f8e4137b66b8343d6dbd7f9be687a41"
server_key_iv: "c4326e0d5f1401fea2338bc89eec013d"
server_key_version: "encryptionkey@0"
I20260929 03:22:31.230435 30844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "3247be068829485e9aa324625b7bf033"
format_stamp: "Formatted at 2026-09-29 03:22:31 on dist-test-slave-fzwk"
server_key: "0f8e4137b66b8343d6dbd7f9be687a41"
server_key_iv: "c4326e0d5f1401fea2338bc89eec013d"
server_key_version: "encryptionkey@0"
I20260929 03:22:31.231911 30844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:31.232702 30859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:31.232892 30844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:31.232998 30844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "3247be068829485e9aa324625b7bf033"
format_stamp: "Formatted at 2026-09-29 03:22:31 on dist-test-slave-fzwk"
server_key: "0f8e4137b66b8343d6dbd7f9be687a41"
server_key_iv: "c4326e0d5f1401fea2338bc89eec013d"
server_key_version: "encryptionkey@0"
I20260929 03:22:31.233078 30844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:31.256330 30844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:31.258563 30844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:31.258740 30844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:31.262992 30844 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46809
I20260929 03:22:31.263031 30911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46809 every 8 connection(s)
I20260929 03:22:31.263594 30844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:31.264323 30912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:31.266760 30912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033: Bootstrap starting.
I20260929 03:22:31.267555 30912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:31.267880 30912 log.cc:824] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:31.268564 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30844
I20260929 03:22:31.268644 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:31.268628 30912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033: No bootstrap required, opened a new log
I20260929 03:22:31.268846 27921 external_mini_cluster.cc:1428] Setting key 25a46b1d9c41a969fcf1fdd39442506b
I20260929 03:22:31.271618 30912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } }
I20260929 03:22:31.271782 30912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:31.271831 30912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3247be068829485e9aa324625b7bf033, State: Initialized, Role: FOLLOWER
I20260929 03:22:31.271989 30912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } }
I20260929 03:22:31.272078 30912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:31.272122 30912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:31.272171 30912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:31.272935 30912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } }
I20260929 03:22:31.273078 30912 leader_election.cc:302] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3247be068829485e9aa324625b7bf033; no voters:
I20260929 03:22:31.273291 30912 leader_election.cc:288] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:31.273317 30917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:31.273479 30917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [term 1 LEADER]: Becoming Leader. State: Replica: 3247be068829485e9aa324625b7bf033, State: Running, Role: LEADER
I20260929 03:22:31.273623 30917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } }
I20260929 03:22:31.273624 30912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:31.274022 30918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3247be068829485e9aa324625b7bf033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } } }
I20260929 03:22:31.274053 30919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3247be068829485e9aa324625b7bf033. Latest consensus state: current_term: 1 leader_uuid: "3247be068829485e9aa324625b7bf033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3247be068829485e9aa324625b7bf033" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46809 } } }
I20260929 03:22:31.274206 30918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:31.274299 30919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:31.274708 30921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:31.275298 30921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:31.277215 30921 catalog_manager.cc:1347] Generated new cluster ID: d16bfa0b4e344a09a0bc92affb0b15b5
I20260929 03:22:31.277278 30921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:31.280889 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:31.269376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.27.68.126:46809)
0929 03:22:31.269533 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:31.269540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:31.269565 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:22:31.269980 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:31.269987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:31.270039 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:31.270181 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:31.271007 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:31.272041 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:31.272976 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:31.273630 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:31.273771 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:31.277203 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:31.277207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:31.277208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:31.277222 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:31.279024 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:31.279488 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:31.279491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:31.279491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:31.279598 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:31.279814 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:31.279815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:31.279815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:31.280080 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:31.280252 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:31.280362 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:22:31.280511 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260929 03:22:31.281798 30921 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:31.282033 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:22:31.282552 30921 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:31.286897 30921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3247be068829485e9aa324625b7bf033: Generated new TSK 0
I20260929 03:22:31.287207 30921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:22:31.297178 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:31.293103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.27.68.126:46809)
0929 03:22:31.293166 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:31.293168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:31.293289 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:22:31.293382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:31.293382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:31.293394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:31.293470 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:31.293902 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:31.294690 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:31.295593 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:31.295816 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:31.295898 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:31.296645 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:31.296646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:31.296646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:31.296862 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:31.296881 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:31.296888 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:31.296955 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:31.297036 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:22:31.297077 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:22:31.299850 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30844
2026-09-29T03:22:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1306 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30945](info): set up 2 sockets
Sep 29 03:22:31 dist-test-slave-fzwk krb5kdc[30945](info): commencing operation
krb5kdc: starting...
W20260929 03:22:32.343508 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:32.464447 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44529
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:32.547953 30958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:32.548187 30958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:32.548211 30958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:32.549438 30958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:32.549486 30958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:32.549500 30958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:32.549513 30958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:32.549526 30958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:32.551062 30958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:32.551357 30958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:32.551602 30958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:32.556576 30966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.556617 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.556744 30963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:32.556739 30958 server_base.cc:1034] running on GCE node
I20260929 03:22:32.557097 30958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:32.557433 30958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:32.558619 30958 hybrid_clock.cc:648] HybridClock initialized: now 1790652152558602 us; error 50 us; skew 500 ppm
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:32.561216 30958 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:32.561682 30958 webserver.cc:460] Webserver started at http://127.27.68.126:32893/ using document root <none> and password file <none>
I20260929 03:22:32.561915 30958 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:32.561986 30958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:32.562145 30958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:32.563292 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "10b76ced908e467a8d5d870571c77b88"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "45ca4e0e5c3bc6073ce42acf8d906660"
server_key_iv: "5a99dc830fe447c6207734a0df600748"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.563691 30958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "10b76ced908e467a8d5d870571c77b88"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "45ca4e0e5c3bc6073ce42acf8d906660"
server_key_iv: "5a99dc830fe447c6207734a0df600748"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.565168 30958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:22:32.565917 30973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.566118 30958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:32.566233 30958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "10b76ced908e467a8d5d870571c77b88"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "45ca4e0e5c3bc6073ce42acf8d906660"
server_key_iv: "5a99dc830fe447c6207734a0df600748"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.566313 30958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:32.578435 30958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:32.580453 30958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:32.580634 30958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:32.584524 30958 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44529
I20260929 03:22:32.584578 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44529 every 8 connection(s)
I20260929 03:22:32.584964 30958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:32.586005 31026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:32.588605 31026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88: Bootstrap starting.
I20260929 03:22:32.589310 31026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:32.589329 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 30958
I20260929 03:22:32.589429 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:32.589659 27921 external_mini_cluster.cc:1428] Setting key 6fe064247611ec2d16ce00e5a7ba4c4a
I20260929 03:22:32.589690 31026 log.cc:824] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:32.590545 31026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88: No bootstrap required, opened a new log
I20260929 03:22:32.592765 31026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } }
I20260929 03:22:32.592909 31026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:32.592933 31026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b76ced908e467a8d5d870571c77b88, State: Initialized, Role: FOLLOWER
I20260929 03:22:32.593107 31026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } }
I20260929 03:22:32.593212 31026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:32.593258 31026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:32.593312 31026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:32.593896 31026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } }
I20260929 03:22:32.594017 31026 leader_election.cc:302] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10b76ced908e467a8d5d870571c77b88; no voters:
I20260929 03:22:32.594203 31026 leader_election.cc:288] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:32.594242 31031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:32.594410 31031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [term 1 LEADER]: Becoming Leader. State: Replica: 10b76ced908e467a8d5d870571c77b88, State: Running, Role: LEADER
I20260929 03:22:32.594480 31026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:32.594611 31031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } }
I20260929 03:22:32.595043 31031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10b76ced908e467a8d5d870571c77b88. Latest consensus state: current_term: 1 leader_uuid: "10b76ced908e467a8d5d870571c77b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } } }
I20260929 03:22:32.595145 31033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10b76ced908e467a8d5d870571c77b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10b76ced908e467a8d5d870571c77b88" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44529 } } }
I20260929 03:22:32.595182 31031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:32.595275 31033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:32.595513 31040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:32.596309 31040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:32.599395 31040 catalog_manager.cc:1347] Generated new cluster ID: cae8d2c180404c2a8e4f9323c5df4275
I20260929 03:22:32.599470 31040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:32.601413 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:32.590201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.27.68.126:44529)
0929 03:22:32.590409 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:32.590419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:32.590456 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:22:32.591024 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:32.591033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:32.591089 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:32.591318 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:32.591990 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.592899 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.593885 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.594561 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.596713 (+ 2152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:32.597652 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:32.597656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:32.597657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:32.597670 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:32.599704 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.600135 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.600138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.600139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.600252 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.600470 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.600471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.600472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.600784 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:32.600935 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:32.601047 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:22:32.601194 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260929 03:22:32.614938 31040 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:32.615581 31040 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:32.619664 31040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10b76ced908e467a8d5d870571c77b88: Generated new TSK 0
I20260929 03:22:32.619879 31040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:32.656080 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:32.740080 31052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:32.740302 31052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:32.740327 31052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:32.741573 31052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:32.741624 31052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:32.741653 31052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:32.743228 31052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:32.743511 31052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:32.743773 31052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:32.748842 31060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.748842 31058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.748879 31057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:32.749027 31052 server_base.cc:1034] running on GCE node
I20260929 03:22:32.749238 31052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:32.749504 31052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:32.750681 31052 hybrid_clock.cc:648] HybridClock initialized: now 1790652152750663 us; error 31 us; skew 500 ppm
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:32.753545 31052 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:32.754048 31052 webserver.cc:460] Webserver started at http://127.27.68.65:35223/ using document root <none> and password file <none>
I20260929 03:22:32.754261 31052 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:32.754305 31052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:32.754400 31052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:32.755515 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "a468eee37a914b12b938442348efc7a9"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "05be49c1e6a85b92d0e6f655747bda6c"
server_key_iv: "e549030590143b074bd2262026e0efbc"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.755842 31052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "a468eee37a914b12b938442348efc7a9"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "05be49c1e6a85b92d0e6f655747bda6c"
server_key_iv: "e549030590143b074bd2262026e0efbc"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.757174 31052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:32.757897 31067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.758111 31052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:32.758198 31052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "a468eee37a914b12b938442348efc7a9"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "05be49c1e6a85b92d0e6f655747bda6c"
server_key_iv: "e549030590143b074bd2262026e0efbc"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.758258 31052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:32.785118 31052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:32.787262 31052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:32.787384 31052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:32.787580 31052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:32.787940 31052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:32.787976 31052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.787999 31052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:32.788058 31052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.793797 31052 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:39547
I20260929 03:22:32.793879 31180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:39547 every 8 connection(s)
I20260929 03:22:32.794251 31052 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:32.801393 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31052
I20260929 03:22:32.801496 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:32.801736 27921 external_mini_cluster.cc:1428] Setting key 2f9463ebcc8271b8faccdc7f5e51f046
I20260929 03:22:32.804061 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:32.795108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:44501 (local address 127.27.68.126:44529)
0929 03:22:32.795191 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:32.795193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:32.795461 (+ 268us) server_negotiation.cc:407] Connection header received
0929 03:22:32.796026 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:32.796028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:32.796037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:32.796092 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:32.797121 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.797622 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.798593 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.798796 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.798848 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:32.800946 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:32.800948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:32.800948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:32.800955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:32.802574 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.802959 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.802960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.802961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.803034 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.803310 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.803311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.803312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.803474 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:32.803542 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:32.803846 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 03:22:32.803945 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:22:32.804348 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:32.795070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44529 (local address 127.27.68.65:44501)
0929 03:22:32.795336 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:32.795374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:32.795930 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:32.796155 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:32.796159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:32.796835 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:32.797037 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:32.797041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.797706 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.797706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:32.798519 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:32.798520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.798871 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.798871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:32.798984 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:32.799500 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:32.799509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:32.800853 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:32.802655 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:32.802662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:32.802664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:32.802885 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:32.803100 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:32.803101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:32.803101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:32.803253 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:32.803536 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:32.803541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:32.803664 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 03:22:32.803788 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 03:22:32.803984 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260929 03:22:32.804849 31181 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44529
I20260929 03:22:32.804966 31181 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:32.805161 31181 heartbeater.cc:511] Master 127.27.68.126:44529 requested a full tablet report, sending...
I20260929 03:22:32.805729 30990 ts_manager.cc:194] Registered new tserver with Master: a468eee37a914b12b938442348efc7a9 (127.27.68.65:39547)
I20260929 03:22:32.806651 30990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:44501
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:32.828042 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:32.908614 31186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:32.908810 31186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:32.908833 31186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:32.909940 31186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:32.909977 31186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:32.910003 31186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:32.911464 31186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:32.911653 31186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:32.911829 31186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:32.916790 31194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.916826 31191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:32.916790 31192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:32.917044 31186 server_base.cc:1034] running on GCE node
I20260929 03:22:32.917263 31186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:32.917462 31186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:32.918599 31186 hybrid_clock.cc:648] HybridClock initialized: now 1790652152918586 us; error 31 us; skew 500 ppm
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:32.921175 31186 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:32.921617 31186 webserver.cc:460] Webserver started at http://127.27.68.66:41529/ using document root <none> and password file <none>
I20260929 03:22:32.921860 31186 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:32.921905 31186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:32.922089 31186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:32.923172 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "ce0d1963c14b467d865a7eebe2baa2ab"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "c30064fab490cbee1a58db7d1484cca3"
server_key_iv: "3a8b3a12edbfc9f72f361d1f857e17c0"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.923609 31186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "ce0d1963c14b467d865a7eebe2baa2ab"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "c30064fab490cbee1a58db7d1484cca3"
server_key_iv: "3a8b3a12edbfc9f72f361d1f857e17c0"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.925045 31186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:32.925828 31201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.926024 31186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:32.926097 31186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "ce0d1963c14b467d865a7eebe2baa2ab"
format_stamp: "Formatted at 2026-09-29 03:22:32 on dist-test-slave-fzwk"
server_key: "c30064fab490cbee1a58db7d1484cca3"
server_key_iv: "3a8b3a12edbfc9f72f361d1f857e17c0"
server_key_version: "encryptionkey@0"
I20260929 03:22:32.926192 31186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:32.938441 31186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:32.940944 31186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:32.941115 31186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:32.941378 31186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:32.941721 31186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:32.941756 31186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.941780 31186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:32.941793 31186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:32.947817 31186 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:33287
I20260929 03:22:32.947896 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:33287 every 8 connection(s)
I20260929 03:22:32.948225 31186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:32.952898 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31186
I20260929 03:22:32.953003 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:32.953227 27921 external_mini_cluster.cc:1428] Setting key e92a4ed09ebae1c43072f1573eaee689
Sep 29 03:22:32 dist-test-slave-fzwk krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652152, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:32.958045 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:32.948981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:41969 (local address 127.27.68.126:44529)
0929 03:22:32.949028 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:32.949030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:32.949222 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:22:32.949704 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:32.949705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:32.949716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:32.949762 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:32.950671 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.951170 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.952217 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.952416 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.952470 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:32.955160 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:32.955162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:32.955163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:32.955172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:32.956606 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.957032 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.957033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.957034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.957100 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:32.957348 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:32.957348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:32.957349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:32.957503 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:32.957574 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:32.957826 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 03:22:32.957924 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:22:32.958071 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:32.948956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44529 (local address 127.27.68.66:41969)
0929 03:22:32.949115 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:32.949151 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:32.949619 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:32.949828 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:32.949832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:32.950400 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:32.950593 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:32.950598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.951324 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.951325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:32.952141 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:32.952143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:32.952488 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:32.952489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:32.952588 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:32.953253 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:32.953263 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:32.955057 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:32.956684 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:32.956691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:32.956693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:32.956946 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:32.957173 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:32.957174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:32.957174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:32.957283 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:32.957566 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:32.957571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:32.957690 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 03:22:32.957775 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:22:32.957902 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260929 03:22:32.958590 31315 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44529
I20260929 03:22:32.958721 31315 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:32.958926 31315 heartbeater.cc:511] Master 127.27.68.126:44529 requested a full tablet report, sending...
I20260929 03:22:32.959426 30990 ts_manager.cc:194] Registered new tserver with Master: ce0d1963c14b467d865a7eebe2baa2ab (127.27.68.66:33287)
I20260929 03:22:32.960008 30990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:41969
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:32.978223 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:33.061586 31320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:33.061874 31320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:33.061904 31320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:33.063234 31320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:33.063284 31320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:33.063328 31320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:33.064885 31320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:33.065186 31320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:33.065472 31320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:33.070794 31326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:33.070785 31328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:33.070788 31325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:33.070990 31320 server_base.cc:1034] running on GCE node
I20260929 03:22:33.071337 31320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:33.071587 31320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:33.072743 31320 hybrid_clock.cc:648] HybridClock initialized: now 1790652153072733 us; error 33 us; skew 500 ppm
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652153, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:33.075487 31320 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:33.076025 31320 webserver.cc:460] Webserver started at http://127.27.68.67:36025/ using document root <none> and password file <none>
I20260929 03:22:33.076272 31320 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:33.076344 31320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:33.076498 31320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:33.077608 31320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "f54e940e75f346258221322d5034d67e"
format_stamp: "Formatted at 2026-09-29 03:22:33 on dist-test-slave-fzwk"
server_key: "c1f7156337dc3c6332ceb4f5254df9ef"
server_key_iv: "702908dc61bdf6036471fa297a6bc6fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:33.077993 31320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "f54e940e75f346258221322d5034d67e"
format_stamp: "Formatted at 2026-09-29 03:22:33 on dist-test-slave-fzwk"
server_key: "c1f7156337dc3c6332ceb4f5254df9ef"
server_key_iv: "702908dc61bdf6036471fa297a6bc6fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:33.079465 31320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:22:33.080205 31335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:33.080379 31320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:33.080485 31320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "f54e940e75f346258221322d5034d67e"
format_stamp: "Formatted at 2026-09-29 03:22:33 on dist-test-slave-fzwk"
server_key: "c1f7156337dc3c6332ceb4f5254df9ef"
server_key_iv: "702908dc61bdf6036471fa297a6bc6fb"
server_key_version: "encryptionkey@0"
I20260929 03:22:33.080567 31320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:33.091970 31320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:33.093894 31320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:33.094066 31320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:33.094317 31320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:33.094635 31320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:33.094691 31320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:33.094732 31320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:33.094760 31320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:33.100983 31320 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:36909
I20260929 03:22:33.101025 31448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:36909 every 8 connection(s)
I20260929 03:22:33.101383 31320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:33.103583 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31320
I20260929 03:22:33.103673 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:33.103842 27921 external_mini_cluster.cc:1428] Setting key ebdd3f491df6164918e49edf0f67d3c5
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652153, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:33.110800 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:33.102116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:46785 (local address 127.27.68.126:44529)
0929 03:22:33.102160 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:33.102163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:33.102369 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:22:33.102907 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:33.102909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:33.102922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:33.102982 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:33.104331 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.104984 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.105956 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.106135 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.106184 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:33.108315 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:33.108316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:33.108317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:33.108324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:33.109646 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:33.110022 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:33.110023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:33.110023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:33.110071 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:33.110276 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:33.110277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:33.110277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:33.110379 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:33.110428 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:33.110673 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:22:33.110734 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:22:33.110983 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:33.102097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44529 (local address 127.27.68.67:46785)
0929 03:22:33.102243 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:33.102281 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:33.102797 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:33.103071 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:33.103074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:33.103899 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:33.104231 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:33.104238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.105067 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.105068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:33.105878 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:33.105880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.106205 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.106205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:33.106308 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:33.106766 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:33.106775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:33.108225 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:33.109734 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:33.109739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:33.109741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:33.109957 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:33.110127 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:33.110128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:33.110128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:33.110223 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:33.110432 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:33.110435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:33.110566 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:22:33.110649 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:33.110768 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260929 03:22:33.111582 31449 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44529
I20260929 03:22:33.111691 31449 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:33.111873 31449 heartbeater.cc:511] Master 127.27.68.126:44529 requested a full tablet report, sending...
I20260929 03:22:33.112278 30990 ts_manager.cc:194] Registered new tserver with Master: f54e940e75f346258221322d5034d67e (127.27.68.67:36909)
I20260929 03:22:33.112653 30990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:46785
I20260929 03:22:33.115533 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[30945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[30945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:33.138814 31030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:33.131037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32844 (local address 127.27.68.126:44529)
0929 03:22:33.131133 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:33.131136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:33.131251 (+ 115us) server_negotiation.cc:407] Connection header received
0929 03:22:33.131344 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:33.131345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:33.131362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:33.131423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:33.132039 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.132561 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.133329 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:33.133525 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:33.133623 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:33.136150 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:33.136152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:33.136153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:33.136163 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:33.137456 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:33.137868 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:33.137869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:33.137870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:33.137941 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:33.138173 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:33.138175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:33.138175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:33.138357 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:33.138485 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:33.138606 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:22:33.138701 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 03:22:33.139125 30990 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32844
I20260929 03:22:33.139649 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31052
I20260929 03:22:33.144205 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31186
I20260929 03:22:33.148550 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31320
I20260929 03:22:33.152925 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 30958
2026-09-29T03:22:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1855 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[31461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[31461](info): set up 2 sockets
Sep 29 03:22:33 dist-test-slave-fzwk krb5kdc[31461](info): commencing operation
krb5kdc: starting...
W20260929 03:22:34.175675 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:34.297564 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43617
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:34.379463 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:34.379737 31474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:34.379776 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:34.381024 31474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:34.381110 31474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:34.381134 31474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:34.381156 31474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:34.381184 31474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:34.382823 31474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:43617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:34.383143 31474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:34.383443 31474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:34.388428 31480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.388422 31479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.388422 31482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:34.388772 31474 server_base.cc:1034] running on GCE node
I20260929 03:22:34.389168 31474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:34.389520 31474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:34.390718 31474 hybrid_clock.cc:648] HybridClock initialized: now 1790652154390711 us; error 43 us; skew 500 ppm
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:34.393553 31474 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:34.394063 31474 webserver.cc:460] Webserver started at http://127.27.68.126:38239/ using document root <none> and password file <none>
I20260929 03:22:34.394320 31474 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:34.394393 31474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:34.394549 31474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:34.395754 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "db16f574be3847ee9e09c11da165fbff"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "2a1cb5bb5849ecc64101c2890e5aa62c"
server_key_iv: "bec87265efe41246cc5de1fab045ae30"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.396153 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "db16f574be3847ee9e09c11da165fbff"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "2a1cb5bb5849ecc64101c2890e5aa62c"
server_key_iv: "bec87265efe41246cc5de1fab045ae30"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.397651 31474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:34.398403 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.398597 31474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:34.398713 31474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "db16f574be3847ee9e09c11da165fbff"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "2a1cb5bb5849ecc64101c2890e5aa62c"
server_key_iv: "bec87265efe41246cc5de1fab045ae30"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.398795 31474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:34.429067 31474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:34.431057 31474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:34.431300 31474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:34.435700 31474 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:43617
I20260929 03:22:34.435750 31541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:43617 every 8 connection(s)
I20260929 03:22:34.436312 31474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:34.436993 31542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:34.439284 31542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff: Bootstrap starting.
I20260929 03:22:34.439831 31542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:34.440176 31542 log.cc:824] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:34.441089 31542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff: No bootstrap required, opened a new log
I20260929 03:22:34.442667 31542 raft_consensus.cc:348] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } }
I20260929 03:22:34.442816 31542 raft_consensus.cc:374] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:34.442836 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31474
I20260929 03:22:34.442839 31542 raft_consensus.cc:729] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db16f574be3847ee9e09c11da165fbff, State: Initialized, Role: FOLLOWER
I20260929 03:22:34.442943 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:34.443004 31542 consensus_queue.cc:260] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } }
I20260929 03:22:34.443114 31542 raft_consensus.cc:388] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:34.443142 27921 external_mini_cluster.cc:1428] Setting key 00369f917263c6ec6b2be8a324708c06
I20260929 03:22:34.443156 31542 raft_consensus.cc:482] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:34.443239 31542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:34.444094 31542 raft_consensus.cc:504] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } }
I20260929 03:22:34.444229 31542 leader_election.cc:302] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db16f574be3847ee9e09c11da165fbff; no voters:
I20260929 03:22:34.444437 31542 leader_election.cc:288] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:34.444464 31547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:34.444571 31547 raft_consensus.cc:686] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [term 1 LEADER]: Becoming Leader. State: Replica: db16f574be3847ee9e09c11da165fbff, State: Running, Role: LEADER
I20260929 03:22:34.444656 31547 consensus_queue.cc:237] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } }
I20260929 03:22:34.444700 31542 sys_catalog.cc:564] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:34.445004 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader db16f574be3847ee9e09c11da165fbff. Latest consensus state: current_term: 1 leader_uuid: "db16f574be3847ee9e09c11da165fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } } }
I20260929 03:22:34.445094 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:34.445019 31548 sys_catalog.cc:455] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db16f574be3847ee9e09c11da165fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db16f574be3847ee9e09c11da165fbff" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43617 } } }
I20260929 03:22:34.445292 31548 sys_catalog.cc:458] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:34.445299 31553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:34.445920 31553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:34.448061 31553 catalog_manager.cc:1347] Generated new cluster ID: eed8285ab9b24e5e977b4690f589b96e
I20260929 03:22:34.448125 31553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:34.455646 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.443664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.27.68.126:43617)
0929 03:22:34.444207 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.444218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.444253 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:22:34.444394 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.444402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.444457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.445415 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.445531 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.446993 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.448006 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.448601 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.448699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.451534 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:34.451540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:34.451540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:34.451554 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:34.453633 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.454014 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.454017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.454018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.454132 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.454301 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.454302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.454303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.454616 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:34.454761 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:34.455253 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 03:22:34.455397 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":457,"threads_started":1}
I20260929 03:22:34.465744 31553 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:34.466346 31553 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:34.471343 31553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db16f574be3847ee9e09c11da165fbff: Generated new TSK 0
I20260929 03:22:34.471570 31553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:34.500222 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:34.587764 31568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:34.588009 31568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:34.588050 31568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:34.589269 31568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:34.589345 31568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:34.589392 31568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:34.590988 31568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:34.591334 31568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:34.591593 31568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:34.596761 31574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:34.596853 31568 server_base.cc:1034] running on GCE node
W20260929 03:22:34.596763 31576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.596763 31573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:34.597224 31568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:34.597505 31568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:34.598678 31568 hybrid_clock.cc:648] HybridClock initialized: now 1790652154598666 us; error 44 us; skew 500 ppm
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:34.601457 31568 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:34.601984 31568 webserver.cc:460] Webserver started at http://127.27.68.65:35437/ using document root <none> and password file <none>
I20260929 03:22:34.602236 31568 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:34.602308 31568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:34.602463 31568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:34.603662 31568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "2c5a9681da344c51a1094771da5ce9bf"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "49207d87fbee7867257f496be50849e5"
server_key_iv: "fe835ec3ca2642cfb13562f7a94fc69c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.604019 31568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "2c5a9681da344c51a1094771da5ce9bf"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "49207d87fbee7867257f496be50849e5"
server_key_iv: "fe835ec3ca2642cfb13562f7a94fc69c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.605446 31568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:34.606158 31583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.606329 31568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:34.606436 31568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "2c5a9681da344c51a1094771da5ce9bf"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "49207d87fbee7867257f496be50849e5"
server_key_iv: "fe835ec3ca2642cfb13562f7a94fc69c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.606510 31568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:34.620004 31568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:34.622458 31568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:34.622634 31568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:34.622876 31568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:34.623265 31568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:34.623330 31568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.623366 31568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:34.623394 31568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.629611 31568 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:36597
I20260929 03:22:34.629652 31696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:36597 every 8 connection(s)
I20260929 03:22:34.629990 31568 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:34.635844 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31568
I20260929 03:22:34.635943 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:34.636132 27921 external_mini_cluster.cc:1428] Setting key 630a57add1c4524d0f556341cf2263cf
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:34.640228 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.630713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43617 (local address 127.27.68.65:40007)
0929 03:22:34.630872 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:34.630915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:34.631918 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:34.632170 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:34.632174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:34.632470 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:34.632679 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.632684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.633362 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.633363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.634184 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.634186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.634534 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.634535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.634637 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.635190 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:34.635199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:34.636999 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:34.638866 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.638873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.638874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.639100 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.639338 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.639338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.639339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.639438 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.639726 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:34.639731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:34.639848 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 03:22:34.639932 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:22:34.640056 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260929 03:22:34.640223 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.630763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40007 (local address 127.27.68.126:43617)
0929 03:22:34.630816 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.630818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.631016 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:22:34.632023 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.632025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.632038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.632108 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.632768 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.633270 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.634269 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.634463 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.634515 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.637120 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:34.637121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:34.637122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:34.637131 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:34.638776 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.639181 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.639182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.639183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.639274 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.639503 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.639504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.639504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.639659 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:34.639730 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:34.639979 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:22:34.640082 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:22:34.640801 31697 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43617
I20260929 03:22:34.640921 31697 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:34.641145 31697 heartbeater.cc:511] Master 127.27.68.126:43617 requested a full tablet report, sending...
I20260929 03:22:34.641707 31506 ts_manager.cc:194] Registered new tserver with Master: 2c5a9681da344c51a1094771da5ce9bf (127.27.68.65:36597)
I20260929 03:22:34.642338 31506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:40007
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:34.662878 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:34.748085 31702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:34.748303 31702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:34.748329 31702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:34.749548 31702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:34.749598 31702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:34.749625 31702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:34.751291 31702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:34.751562 31702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:34.751814 31702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:22:34.757102 31702 server_base.cc:1034] running on GCE node
W20260929 03:22:34.757117 31707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.757125 31710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.757120 31708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:34.757625 31702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:34.757882 31702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:34.759073 31702 hybrid_clock.cc:648] HybridClock initialized: now 1790652154759045 us; error 42 us; skew 500 ppm
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:34.761749 31702 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:34.762363 31702 webserver.cc:460] Webserver started at http://127.27.68.66:38441/ using document root <none> and password file <none>
I20260929 03:22:34.762650 31702 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:34.762724 31702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:34.762866 31702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:34.764246 31702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "e6210c2a80bc42eeb68bfaaffaa72c90"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "0b2bb5d83804b313b6dc33a1d8c86f57"
server_key_iv: "47b0b594e8ae34d6e565cca23b411b8c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.764624 31702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "e6210c2a80bc42eeb68bfaaffaa72c90"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "0b2bb5d83804b313b6dc33a1d8c86f57"
server_key_iv: "47b0b594e8ae34d6e565cca23b411b8c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.766004 31702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:22:34.766844 31717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.767010 31702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:34.767127 31702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "e6210c2a80bc42eeb68bfaaffaa72c90"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "0b2bb5d83804b313b6dc33a1d8c86f57"
server_key_iv: "47b0b594e8ae34d6e565cca23b411b8c"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.767204 31702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:34.784732 31702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:34.786968 31702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:34.787173 31702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:34.787457 31702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:34.787796 31702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:34.787853 31702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.787901 31702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:34.787925 31702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.793893 31702 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:39357
I20260929 03:22:34.793934 31830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:39357 every 8 connection(s)
I20260929 03:22:34.794292 31702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:34.798177 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31702
I20260929 03:22:34.798283 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:34.798456 27921 external_mini_cluster.cc:1428] Setting key 21019ff2122e99399cf6198bf2e2457d
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:34.804908 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.795095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43617 (local address 127.27.68.66:47489)
0929 03:22:34.795296 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:34.795342 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:34.796255 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:34.796532 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:34.796536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:34.796800 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:34.797001 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.797004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.797658 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.797658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.798768 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.798771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.799239 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.799241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.799393 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.800002 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:34.800011 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:34.801438 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:34.803470 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.803477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.803478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.803723 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.803956 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.803957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.803957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.804062 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.804377 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:34.804381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:34.804504 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 03:22:34.804598 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:22:34.804730 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260929 03:22:34.804903 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.795094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:47489 (local address 127.27.68.126:43617)
0929 03:22:34.795162 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.795164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.795434 (+ 270us) server_negotiation.cc:407] Connection header received
0929 03:22:34.796375 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.796376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.796388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.796450 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.797075 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.797582 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.798870 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.799130 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.799209 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.801546 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:34.801548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:34.801549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:34.801558 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:34.803389 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.803808 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.803809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.803810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.803881 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.804130 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.804131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.804132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.804303 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:34.804386 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:34.804649 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 03:22:34.804763 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:22:34.805635 31831 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43617
I20260929 03:22:34.805755 31831 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:34.805950 31831 heartbeater.cc:511] Master 127.27.68.126:43617 requested a full tablet report, sending...
I20260929 03:22:34.806413 31506 ts_manager.cc:194] Registered new tserver with Master: e6210c2a80bc42eeb68bfaaffaa72c90 (127.27.68.66:39357)
I20260929 03:22:34.806973 31506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:47489
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:34.825567 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:34.907090 31836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:34.907387 31836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:34.907430 31836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:34.908725 31836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:34.908790 31836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:34.908835 31836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:34.910501 31836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:34.910796 31836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:34.911048 31836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:34.916268 31841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.916268 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:34.916268 31842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:34.916515 31836 server_base.cc:1034] running on GCE node
I20260929 03:22:34.916807 31836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:34.917119 31836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:34.918296 31836 hybrid_clock.cc:648] HybridClock initialized: now 1790652154918281 us; error 48 us; skew 500 ppm
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:34.920923 31836 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:34.921444 31836 webserver.cc:460] Webserver started at http://127.27.68.67:40915/ using document root <none> and password file <none>
I20260929 03:22:34.921712 31836 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:34.921783 31836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:34.921937 31836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:34.923027 31836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "69c5108aac9f4d509f2a6ee6dfb3a808"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "d2fb7fc92e50d67c08674d2d08feca05"
server_key_iv: "5e229546313832d4624582b980f6506a"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.923420 31836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "69c5108aac9f4d509f2a6ee6dfb3a808"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "d2fb7fc92e50d67c08674d2d08feca05"
server_key_iv: "5e229546313832d4624582b980f6506a"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.924789 31836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:34.925572 31851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.925771 31836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:34.925877 31836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "69c5108aac9f4d509f2a6ee6dfb3a808"
format_stamp: "Formatted at 2026-09-29 03:22:34 on dist-test-slave-fzwk"
server_key: "d2fb7fc92e50d67c08674d2d08feca05"
server_key_iv: "5e229546313832d4624582b980f6506a"
server_key_version: "encryptionkey@0"
I20260929 03:22:34.925956 31836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:34.945303 31836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:34.947786 31836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:34.947949 31836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:34.948196 31836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:34.948518 31836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:34.948558 31836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.948616 31836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:34.948644 31836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:34.954443 31836 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:44261
I20260929 03:22:34.954525 31964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:44261 every 8 connection(s)
I20260929 03:22:34.954860 31836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:34.960745 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31836
I20260929 03:22:34.960832 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:34.961055 27921 external_mini_cluster.cc:1428] Setting key f8d155e3047afc56224d670722d4e02f
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:34.964814 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.955678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:55609 (local address 127.27.68.126:43617)
0929 03:22:34.955738 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.955740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.956038 (+ 298us) server_negotiation.cc:407] Connection header received
0929 03:22:34.957007 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.957008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.957019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.957073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.957831 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.958318 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.959446 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.959656 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.959719 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.962108 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:34.962110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:34.962111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:34.962127 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:34.963369 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.963803 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.963805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.963805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.963873 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.964138 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.964139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.964139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.964269 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:34.964347 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:34.964629 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 03:22:34.964716 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:22:34.965027 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.955661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43617 (local address 127.27.68.67:55609)
0929 03:22:34.955872 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:34.955928 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:34.956911 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:34.957139 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:34.957143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:34.957493 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:34.957751 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.957756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.958408 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.958409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.959348 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:34.959350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.959735 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.959735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:34.959835 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.960328 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:34.960336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:34.962008 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:34.963452 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.963460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.963462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.963722 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.963940 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:34.963941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:34.963941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:34.964067 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:34.964337 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:34.964344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:34.964495 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:22:34.964607 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:22:34.964781 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260929 03:22:34.965536 31965 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43617
I20260929 03:22:34.965652 31965 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:34.965839 31965 heartbeater.cc:511] Master 127.27.68.126:43617 requested a full tablet report, sending...
I20260929 03:22:34.966358 31506 ts_manager.cc:194] Registered new tserver with Master: 69c5108aac9f4d509f2a6ee6dfb3a808 (127.27.68.67:44261)
I20260929 03:22:34.966766 31506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:55609
I20260929 03:22:34.973040 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:34.996250 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.988487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.27.68.65:36597)
0929 03:22:34.988662 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.988667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.988676 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:34.988702 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.988703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.988716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.988804 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.989403 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.990136 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.990925 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.991307 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.991398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:34.993724 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:34.993732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:34.993732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:34.993738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:34.995182 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.995581 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.995584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.995584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.995638 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:34.995799 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:34.995800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:34.995800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:34.995928 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:34.996043 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:34.996104 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:22:34.996169 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
Sep 29 03:22:34 dist-test-slave-fzwk krb5kdc[31461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:35.004737 31546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:34.997634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.27.68.126:43617)
0929 03:22:34.997705 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:34.997707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:34.997717 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:34.997753 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:34.997754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:34.997764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:34.997829 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:34.998311 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.998800 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.999549 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:34.999744 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:34.999840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:35.002172 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:35.002173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:35.002174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:35.002182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:35.003512 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:35.003875 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:35.003876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:35.003877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:35.003934 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:35.004130 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:35.004131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:35.004131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:35.004293 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:35.004399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:35.004540 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:22:35.004623 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 03:22:35.004987 31506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58256
I20260929 03:22:35.005409 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31568
I20260929 03:22:35.010250 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31702
I20260929 03:22:35.014834 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31836
I20260929 03:22:35.019029 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31474
2026-09-29T03:22:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1866 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:35 dist-test-slave-fzwk krb5kdc[31977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:35 dist-test-slave-fzwk krb5kdc[31977](info): set up 2 sockets
Sep 29 03:22:35 dist-test-slave-fzwk krb5kdc[31977](info): commencing operation
krb5kdc: starting...
W20260929 03:22:36.055521 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:36.174117 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42925
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:36.257225 31990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:36.257508 31990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:36.257560 31990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:36.258847 31990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:36.258908 31990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:36.258931 31990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:36.258960 31990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:36.258989 31990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:36.260753 31990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:36.261057 31990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:36.261296 31990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:36.266309 31996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.266340 31998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.266387 31990 server_base.cc:1034] running on GCE node
W20260929 03:22:36.266309 31995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.266821 31990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:36.267145 31990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:36.268342 31990 hybrid_clock.cc:648] HybridClock initialized: now 1790652156268333 us; error 33 us; skew 500 ppm
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:36.270876 31990 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:36.271302 31990 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:36.271375 31990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:36.271531 31990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:36.272655 31990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "4fc71a2a93514036ac476d67651e5ce2"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "392e5fcce98802109a534e44df74995b"
server_key_iv: "cee1f93ceefe094b612aa931bd32f8ad"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.273025 31990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "4fc71a2a93514036ac476d67651e5ce2"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "392e5fcce98802109a534e44df74995b"
server_key_iv: "cee1f93ceefe094b612aa931bd32f8ad"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.274353 31990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:36.275044 32004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.275267 31990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.275373 31990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "4fc71a2a93514036ac476d67651e5ce2"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "392e5fcce98802109a534e44df74995b"
server_key_iv: "cee1f93ceefe094b612aa931bd32f8ad"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.275462 31990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:36.296028 31990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:36.298588 31990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:36.298785 31990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:36.302794 31990 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42925
I20260929 03:22:36.302836 32056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42925 every 8 connection(s)
I20260929 03:22:36.303155 31990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:36.304165 32057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:36.306495 32057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: Bootstrap starting.
I20260929 03:22:36.307083 32057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:36.307435 32057 log.cc:824] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:36.308126 32057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: No bootstrap required, opened a new log
I20260929 03:22:36.309615 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 31990
I20260929 03:22:36.309726 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:36.309911 27921 external_mini_cluster.cc:1428] Setting key 130475e6c3a2283ab079646ef55eb371
I20260929 03:22:36.310048 32057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } }
I20260929 03:22:36.310230 32057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:36.310254 32057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc71a2a93514036ac476d67651e5ce2, State: Initialized, Role: FOLLOWER
I20260929 03:22:36.310400 32057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } }
I20260929 03:22:36.310487 32057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:36.310531 32057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:36.310592 32057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:36.311480 32057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } }
I20260929 03:22:36.311597 32057 leader_election.cc:302] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc71a2a93514036ac476d67651e5ce2; no voters:
I20260929 03:22:36.311770 32057 leader_election.cc:288] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:36.311800 32062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:36.311902 32062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [term 1 LEADER]: Becoming Leader. State: Replica: 4fc71a2a93514036ac476d67651e5ce2, State: Running, Role: LEADER
I20260929 03:22:36.311988 32062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } }
I20260929 03:22:36.312063 32057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:36.312345 32062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc71a2a93514036ac476d67651e5ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } } }
I20260929 03:22:36.312417 32062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:36.312549 32062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc71a2a93514036ac476d67651e5ce2. Latest consensus state: current_term: 1 leader_uuid: "4fc71a2a93514036ac476d67651e5ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc71a2a93514036ac476d67651e5ce2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42925 } } }
I20260929 03:22:36.312609 32062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:36.313090 32066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:36.313578 32066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:36.315730 32066 catalog_manager.cc:1347] Generated new cluster ID: 7b79d8ff8f624f9a90e7dab1f8f69999
I20260929 03:22:36.316079 32066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:36.322590 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.310470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.27.68.126:42925)
0929 03:22:36.310680 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.310691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.310727 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:22:36.311168 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.311178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.311281 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.311457 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:36.312304 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.313742 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.314876 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.315421 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.315541 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.318381 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:36.318387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:36.318387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:36.318401 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:36.320528 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.320941 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.320945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.320945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.321066 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.321253 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.321254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.321254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.321571 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:36.321703 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.322199 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.322350 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260929 03:22:36.330452 32066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:36.331039 32066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:36.348709 32066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: Generated new TSK 0
I20260929 03:22:36.348944 32066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:36.382617 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:36.467881 32083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:36.468096 32083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:36.468122 32083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:36.469273 32083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:36.469311 32083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:36.469337 32083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:36.470837 32083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:36.471038 32083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:36.471313 32083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:36.476222 32088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.476243 32089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.476240 32091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.476470 32083 server_base.cc:1034] running on GCE node
I20260929 03:22:36.476706 32083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:36.476965 32083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:36.478140 32083 hybrid_clock.cc:648] HybridClock initialized: now 1790652156478124 us; error 46 us; skew 500 ppm
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:36.480883 32083 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:36.481375 32083 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:36.481462 32083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:36.481608 32083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:36.482777 32083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "2e4bacc3d3a5499fa3f04edac553a333"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "3ae1dbeeea539ce72da47b7b70272efa"
server_key_iv: "23472593661606614b1b5b0121bbe4d0"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.483167 32083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "2e4bacc3d3a5499fa3f04edac553a333"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "3ae1dbeeea539ce72da47b7b70272efa"
server_key_iv: "23472593661606614b1b5b0121bbe4d0"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.484627 32083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:36.485376 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.485555 32083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:36.485664 32083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "2e4bacc3d3a5499fa3f04edac553a333"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "3ae1dbeeea539ce72da47b7b70272efa"
server_key_iv: "23472593661606614b1b5b0121bbe4d0"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.485764 32083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:36.503777 32083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:36.506253 32083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:36.506433 32083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:36.506682 32083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:36.507015 32083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:36.507069 32083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.507110 32083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:36.507138 32083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.512933 32083 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:38753
I20260929 03:22:36.512974 32210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:38753 every 8 connection(s)
I20260929 03:22:36.513338 32083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:36.518013 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32083
I20260929 03:22:36.518146 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:36.518379 27921 external_mini_cluster.cc:1428] Setting key 10cbf1c4c079b6cd078e51515a0d04d0
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:36.526154 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.514103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34841 (local address 127.27.68.126:42925)
0929 03:22:36.514150 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.514152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.514391 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:22:36.514865 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.514866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.514879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.514930 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:36.516460 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.517357 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.519421 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.519703 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.519777 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.522925 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:36.522928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:36.522928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:36.522937 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:36.524667 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.525097 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.525099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.525100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.525171 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.525413 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.525414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.525415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.525585 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:36.525668 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.525936 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.526032 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:22:36.526182 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.514086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42925 (local address 127.27.68.65:34841)
0929 03:22:36.514247 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.514302 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.514779 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.515005 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.515008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.515480 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:36.516325 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.516334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.517480 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.517481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.519335 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.519339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.519805 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.519806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.519997 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.520676 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:36.520696 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:36.522778 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:36.524772 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.524779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.524781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.525016 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.525243 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.525244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.525244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.525345 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.525657 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:36.525661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:36.525796 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.525883 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.526012 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260929 03:22:36.527037 32211 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42925
I20260929 03:22:36.527129 32211 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:36.527334 32211 heartbeater.cc:511] Master 127.27.68.126:42925 requested a full tablet report, sending...
I20260929 03:22:36.527998 32021 ts_manager.cc:194] Registered new tserver with Master: 2e4bacc3d3a5499fa3f04edac553a333 (127.27.68.65:38753)
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:22:36.528944 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:34841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:36.546068 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:36.627936 32216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:36.628157 32216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:36.628181 32216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:36.629390 32216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:36.629439 32216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:36.629467 32216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:36.631006 32216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:36.631305 32216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:36.631552 32216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:36.636420 32221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.636580 32224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.636613 32216 server_base.cc:1034] running on GCE node
W20260929 03:22:36.636652 32222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.636894 32216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:36.637173 32216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:36.638348 32216 hybrid_clock.cc:648] HybridClock initialized: now 1790652156638333 us; error 45 us; skew 500 ppm
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:36.641067 32216 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:36.641491 32216 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:36.641548 32216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:36.641741 32216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:36.642926 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "d5118ae04d924027811f1fc6bac00b31"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "6bc89cdd5662d016cb337c331acdad0a"
server_key_iv: "15a88724e934780fcc7655c824b72305"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.643370 32216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "d5118ae04d924027811f1fc6bac00b31"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "6bc89cdd5662d016cb337c331acdad0a"
server_key_iv: "15a88724e934780fcc7655c824b72305"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.644815 32216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:36.645614 32230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.645814 32216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:36.645916 32216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "d5118ae04d924027811f1fc6bac00b31"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "6bc89cdd5662d016cb337c331acdad0a"
server_key_iv: "15a88724e934780fcc7655c824b72305"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.645999 32216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:36.663520 32216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:36.665566 32216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:36.665753 32216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:36.666030 32216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:36.666414 32216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:36.666451 32216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.666499 32216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:36.666531 32216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.673147 32216 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:46617
I20260929 03:22:36.673182 32343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:46617 every 8 connection(s)
I20260929 03:22:36.673513 32216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:36.681591 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32216
I20260929 03:22:36.681707 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:36.681922 27921 external_mini_cluster.cc:1428] Setting key 41e2b6f77c48fa3ce119561930e78720
I20260929 03:22:36.683826 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.674318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:36571 (local address 127.27.68.126:42925)
0929 03:22:36.674382 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.674384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.674614 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:22:36.675152 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.675154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.675167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.675233 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:36.676232 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.676753 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.677895 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.678098 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.678210 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.680531 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:36.680532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:36.680533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:36.680541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:36.682084 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.682603 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.682605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.682605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.682688 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.683022 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.683023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.683024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.683171 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:36.683304 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.683675 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.683758 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:36.684135 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.674318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42925 (local address 127.27.68.66:36571)
0929 03:22:36.674492 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.674529 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.675036 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.675323 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.675326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.675598 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:36.676106 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.676111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.676860 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.676860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.677806 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.677809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.678220 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.678221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.678353 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.678943 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:36.678952 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:36.680418 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:36.682160 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.682168 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.682171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.682457 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.682769 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.682769 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.682770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.682946 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.683299 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:36.683306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:36.683498 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.683624 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.683825 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260929 03:22:36.684887 32344 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42925
I20260929 03:22:36.684978 32344 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:36.685149 32344 heartbeater.cc:511] Master 127.27.68.126:42925 requested a full tablet report, sending...
I20260929 03:22:36.685563 32021 ts_manager.cc:194] Registered new tserver with Master: d5118ae04d924027811f1fc6bac00b31 (127.27.68.66:46617)
I20260929 03:22:36.685954 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:36571
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:36.707947 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:36.790639 32349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:36.790879 32349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:36.790905 32349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:36.792093 32349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:36.792136 32349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:36.792165 32349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:36.793782 32349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:36.793991 32349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:36.794183 32349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:36.799163 32355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.799188 32354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:36.799156 32357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:36.799564 32349 server_base.cc:1034] running on GCE node
I20260929 03:22:36.799801 32349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:36.800098 32349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:36.801272 32349 hybrid_clock.cc:648] HybridClock initialized: now 1790652156801269 us; error 37 us; skew 500 ppm
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:36.803975 32349 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:36.804380 32349 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:36.804453 32349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:36.804616 32349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:36.805699 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "c84ae2c896b04310a19103b74a409f60"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "f1072c36e702091d8c1d2bbd8c60eb09"
server_key_iv: "dad1a4bc847237200150cd1ce800f913"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.806077 32349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "c84ae2c896b04310a19103b74a409f60"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "f1072c36e702091d8c1d2bbd8c60eb09"
server_key_iv: "dad1a4bc847237200150cd1ce800f913"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.807487 32349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:36.808249 32363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.808422 32349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:36.808521 32349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "c84ae2c896b04310a19103b74a409f60"
format_stamp: "Formatted at 2026-09-29 03:22:36 on dist-test-slave-fzwk"
server_key: "f1072c36e702091d8c1d2bbd8c60eb09"
server_key_iv: "dad1a4bc847237200150cd1ce800f913"
server_key_version: "encryptionkey@0"
I20260929 03:22:36.808606 32349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:36.826558 32349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:36.828594 32349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:36.828779 32349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:36.829025 32349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:36.829350 32349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:36.829406 32349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.829449 32349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:36.829474 32349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:36.835692 32349 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:34293
I20260929 03:22:36.835716 32476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:34293 every 8 connection(s)
I20260929 03:22:36.836035 32349 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:22:36 dist-test-slave-fzwk krb5kdc[31977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652156, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:36.843155 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32349
I20260929 03:22:36.843290 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:36.843462 27921 external_mini_cluster.cc:1428] Setting key db2d061ccd282337a6370197a64ac123
I20260929 03:22:36.846197 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.836762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:37581 (local address 127.27.68.126:42925)
0929 03:22:36.836807 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.836809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.837049 (+ 240us) server_negotiation.cc:407] Connection header received
0929 03:22:36.837617 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.837619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.837632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.837702 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:36.838723 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.839345 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.840392 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.840598 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.840677 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.843042 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:36.843043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:36.843044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:36.843052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:36.844737 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.845162 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.845163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.845164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.845231 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.845494 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.845495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.845496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.845633 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:36.845712 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.846003 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.846097 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:36.846414 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.836749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42925 (local address 127.27.68.67:37581)
0929 03:22:36.836915 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.836955 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.837483 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.837802 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.837806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.838086 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:36.838610 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.838615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.839440 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.839441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.840296 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.840298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.840684 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.840686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.840815 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.841378 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:36.841387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:36.842918 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:36.844823 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.844831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.844833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.845082 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.845299 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:36.845300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:36.845301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:36.845423 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:36.845702 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:36.845708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:36.845864 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.845980 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.846163 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260929 03:22:36.846925 32477 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42925
I20260929 03:22:36.847029 32477 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:36.847193 32477 heartbeater.cc:511] Master 127.27.68.126:42925 requested a full tablet report, sending...
I20260929 03:22:36.847618 32021 ts_manager.cc:194] Registered new tserver with Master: c84ae2c896b04310a19103b74a409f60 (127.27.68.67:34293)
I20260929 03:22:36.847999 32021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:37581
I20260929 03:22:36.855309 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:22:36.862321 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.856646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.27.68.126:42925)
0929 03:22:36.856708 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.856710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.856793 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:22:36.856834 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.856835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.856844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.856889 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:36.857392 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.857869 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.858552 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.858742 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.858791 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.860030 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:36.860031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:36.860032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:36.860037 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:36.861235 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.861591 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.861592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.861593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.861656 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:36.861831 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:36.861831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:36.861831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:36.861979 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:36.862057 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.862159 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.862222 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:36.864135 32021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:36.865782 32021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:36.872474 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:46617 (local address 127.0.0.1:38086)
0929 03:22:36.868508 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.868518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.868571 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.868925 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.868926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.868928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:36.868967 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.868969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.870260 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.870261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.871393 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.871395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.872244 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.872245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.872302 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.872306 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.872354 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.872382 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260929 03:22:36.872483 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.27.68.66:46617)
0929 03:22:36.868757 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.868761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.868771 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:36.868808 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.868810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.868825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.869042 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:36.869116 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.870153 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.871466 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.872145 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.872213 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.872281 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.872364 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.872402 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260929 03:22:36.872561 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.27.68.65:38753)
0929 03:22:36.868619 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.868624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.868634 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:36.868675 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.868677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.868694 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.868771 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:36.869029 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.869812 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.871311 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.872036 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.872116 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.872184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.872431 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.872471 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":73,"threads_started":1}
I20260929 03:22:36.873004 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38753 (local address 127.0.0.1:33910)
0929 03:22:36.868439 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.868448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.868491 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.868864 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.868866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.868868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:36.868942 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.868945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.869917 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.869919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.871172 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.871174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.872123 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.872123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.872214 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.872247 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.872315 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.872379 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260929 03:22:36.873279 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34293 (local address 127.0.0.1:37966)
0929 03:22:36.868344 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.868392 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.868467 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.868813 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.868816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.868820 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:36.868913 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.868916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.870282 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.870283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.871163 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.871166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.873106 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.873107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.873167 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.873170 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.873208 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.873237 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260929 03:22:36.873360 32278 tablet_service.cc:1467] Processing CreateTablet for tablet a0947ceda19648d093109df0728e10de (DEFAULT_TABLE table=test-table [id=8b168cacc0744674b6e23187a25698ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:36.873587 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.868255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.27.68.67:34293)
0929 03:22:36.868618 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.868624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.868634 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:36.868678 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.868680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.868694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.868803 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:36.869001 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.870154 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.872215 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.873008 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.873103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.873151 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.873470 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.873511 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":72,"threads_started":1}
I20260929 03:22:36.873746 32278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0947ceda19648d093109df0728e10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:36.873996 32145 tablet_service.cc:1467] Processing CreateTablet for tablet a0947ceda19648d093109df0728e10de (DEFAULT_TABLE table=test-table [id=8b168cacc0744674b6e23187a25698ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:36.874308 32145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0947ceda19648d093109df0728e10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:36.875910 32411 tablet_service.cc:1467] Processing CreateTablet for tablet a0947ceda19648d093109df0728e10de (DEFAULT_TABLE table=test-table [id=8b168cacc0744674b6e23187a25698ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:36.876226 32411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0947ceda19648d093109df0728e10de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:36.876775 32494 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Bootstrap starting.
I20260929 03:22:36.877368 32494 tablet_bootstrap.cc:654] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:36.877418 32495 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: Bootstrap starting.
I20260929 03:22:36.877988 32494 log.cc:824] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:36.878113 32495 tablet_bootstrap.cc:654] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:36.878739 32495 log.cc:824] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:36.878849 32494 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: No bootstrap required, opened a new log
I20260929 03:22:36.878908 32494 ts_tablet_manager.cc:1397] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:36.879374 32498 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Bootstrap starting.
I20260929 03:22:36.879546 32495 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: No bootstrap required, opened a new log
I20260929 03:22:36.879621 32495 ts_tablet_manager.cc:1397] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:36.880018 32498 tablet_bootstrap.cc:654] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:36.880601 32498 log.cc:824] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:36.881392 32498 tablet_bootstrap.cc:492] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: No bootstrap required, opened a new log
I20260929 03:22:36.881469 32498 ts_tablet_manager.cc:1397] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:36.881364 32495 raft_consensus.cc:348] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.881542 32495 raft_consensus.cc:374] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:36.881590 32495 raft_consensus.cc:729] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4bacc3d3a5499fa3f04edac553a333, State: Initialized, Role: FOLLOWER
I20260929 03:22:36.881584 32494 raft_consensus.cc:348] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.881764 32494 raft_consensus.cc:374] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:36.881711 32495 consensus_queue.cc:260] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.881808 32494 raft_consensus.cc:729] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5118ae04d924027811f1fc6bac00b31, State: Initialized, Role: FOLLOWER
I20260929 03:22:36.881932 32494 consensus_queue.cc:260] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.882012 32495 ts_tablet_manager.cc:1428] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:22:36.882012 32211 heartbeater.cc:503] Master 127.27.68.126:42925 was elected leader, sending a full tablet report...
I20260929 03:22:36.882193 32494 ts_tablet_manager.cc:1428] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:36.882318 32344 heartbeater.cc:503] Master 127.27.68.126:42925 was elected leader, sending a full tablet report...
I20260929 03:22:36.883606 32498 raft_consensus.cc:348] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.883773 32498 raft_consensus.cc:374] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:36.883821 32498 raft_consensus.cc:729] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84ae2c896b04310a19103b74a409f60, State: Initialized, Role: FOLLOWER
I20260929 03:22:36.883940 32498 consensus_queue.cc:260] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.884227 32498 ts_tablet_manager.cc:1428] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:36.884258 32477 heartbeater.cc:503] Master 127.27.68.126:42925 was elected leader, sending a full tablet report...
W20260929 03:22:36.924134 32345 tablet.cc:2369] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:36.954262 32500 raft_consensus.cc:482] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:36.954450 32500 raft_consensus.cc:504] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.954841 32500 leader_election.cc:288] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c84ae2c896b04310a19103b74a409f60 (127.27.68.67:34293), d5118ae04d924027811f1fc6bac00b31 (127.27.68.66:46617)
I20260929 03:22:36.958182 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.955184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:49907 (local address 127.27.68.67:34293)
0929 03:22:36.955279 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.955282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.955295 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:22:36.955329 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.955330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.955341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.955403 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:36.955580 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.956345 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957402 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.957812 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957903 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.957912 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.958071 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.958110 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 03:22:36.958225 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.955073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:34293 (local address 127.27.68.65:49907)
0929 03:22:36.955136 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.955150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.955265 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.955420 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.955421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.955424 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:36.955501 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.955503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.956477 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.956478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.957287 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.957289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.957914 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.957996 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.957999 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.958065 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.958106 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:22:36.958242 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.955343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38641 (local address 127.27.68.66:46617)
0929 03:22:36.955422 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.955424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.955464 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:22:36.955583 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.955584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.955593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.955653 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:36.955796 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.956489 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957492 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.957874 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957962 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.957971 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.958133 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.958171 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:36.958338 32503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.955198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:46617 (local address 127.27.68.65:38641)
0929 03:22:36.955411 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:36.955420 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:36.955488 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:36.955664 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:36.955664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:36.955666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:36.955726 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.955727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.956591 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.956592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.957346 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:36.957347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.957963 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.957964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:36.958048 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.958051 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:36.958116 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:22:36.958158 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":143,"threads_started":1}
I20260929 03:22:36.958622 32298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0947ceda19648d093109df0728e10de" candidate_uuid: "2e4bacc3d3a5499fa3f04edac553a333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5118ae04d924027811f1fc6bac00b31" is_pre_election: true
I20260929 03:22:36.958637 32431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0947ceda19648d093109df0728e10de" candidate_uuid: "2e4bacc3d3a5499fa3f04edac553a333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ae2c896b04310a19103b74a409f60" is_pre_election: true
I20260929 03:22:36.958814 32431 raft_consensus.cc:2454] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4bacc3d3a5499fa3f04edac553a333 in term 0.
I20260929 03:22:36.958819 32298 raft_consensus.cc:2454] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4bacc3d3a5499fa3f04edac553a333 in term 0.
I20260929 03:22:36.959084 32101 leader_election.cc:302] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4bacc3d3a5499fa3f04edac553a333, c84ae2c896b04310a19103b74a409f60; no voters:
I20260929 03:22:36.959242 32500 raft_consensus.cc:2781] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:36.959309 32500 raft_consensus.cc:482] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:36.959326 32500 raft_consensus.cc:3037] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:36.960219 32500 raft_consensus.cc:504] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.960362 32500 leader_election.cc:288] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [CANDIDATE]: Term 1 election: Requested vote from peers c84ae2c896b04310a19103b74a409f60 (127.27.68.67:34293), d5118ae04d924027811f1fc6bac00b31 (127.27.68.66:46617)
I20260929 03:22:36.960551 32298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0947ceda19648d093109df0728e10de" candidate_uuid: "2e4bacc3d3a5499fa3f04edac553a333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5118ae04d924027811f1fc6bac00b31"
I20260929 03:22:36.960577 32431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0947ceda19648d093109df0728e10de" candidate_uuid: "2e4bacc3d3a5499fa3f04edac553a333" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c84ae2c896b04310a19103b74a409f60"
I20260929 03:22:36.960630 32298 raft_consensus.cc:3037] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:36.960669 32431 raft_consensus.cc:3037] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:36.961402 32431 raft_consensus.cc:2454] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4bacc3d3a5499fa3f04edac553a333 in term 1.
I20260929 03:22:36.961404 32298 raft_consensus.cc:2454] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4bacc3d3a5499fa3f04edac553a333 in term 1.
I20260929 03:22:36.961585 32101 leader_election.cc:302] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4bacc3d3a5499fa3f04edac553a333, d5118ae04d924027811f1fc6bac00b31; no voters:
I20260929 03:22:36.961674 32500 raft_consensus.cc:2781] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:36.961836 32500 raft_consensus.cc:686] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [term 1 LEADER]: Becoming Leader. State: Replica: 2e4bacc3d3a5499fa3f04edac553a333, State: Running, Role: LEADER
I20260929 03:22:36.961962 32500 consensus_queue.cc:237] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } }
I20260929 03:22:36.962832 32020 catalog_manager.cc:5495] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4bacc3d3a5499fa3f04edac553a333 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "2e4bacc3d3a5499fa3f04edac553a333" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4bacc3d3a5499fa3f04edac553a333" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:36.987848 32491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:36.985466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.27.68.65:38753)
0929 03:22:36.985532 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:36.985534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:36.985545 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:36.985586 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:36.985587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:36.985605 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:36.985671 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:36.985808 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.986280 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.987027 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:36.987272 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:36.987335 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:36.987477 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:36.987588 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:36.987652 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:36.987683 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 03:22:36.987702 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:22:36.989683 32431 raft_consensus.cc:1261] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 1 FOLLOWER]: Refusing update from remote peer 2e4bacc3d3a5499fa3f04edac553a333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:36.989683 32298 raft_consensus.cc:1261] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 1 FOLLOWER]: Refusing update from remote peer 2e4bacc3d3a5499fa3f04edac553a333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:36.990061 32504 consensus_queue.cc:1035] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c84ae2c896b04310a19103b74a409f60" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:36.990197 32500 consensus_queue.cc:1035] T a0947ceda19648d093109df0728e10de P 2e4bacc3d3a5499fa3f04edac553a333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5118ae04d924027811f1fc6bac00b31" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 46617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:36.992444 32507 mvcc.cc:205] Tried to move back new op lower bound from 7334511235026960384 to 7334511234916724736. Current Snapshot: MvccSnapshot[applied={T|T < 7334511235026960384}]
I20260929 03:22:36.992686 32508 mvcc.cc:205] Tried to move back new op lower bound from 7334511235026960384 to 7334511234916724736. Current Snapshot: MvccSnapshot[applied={T|T < 7334511235026960384}]
I20260929 03:22:36.992933 32509 mvcc.cc:205] Tried to move back new op lower bound from 7334511235026960384 to 7334511234916724736. Current Snapshot: MvccSnapshot[applied={T|T < 7334511235026960384}]
I20260929 03:22:36.995179 32021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:36.995311 32021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:36.996138 32021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fc71a2a93514036ac476d67651e5ce2: Sending DeleteTablet for 3 replicas of tablet a0947ceda19648d093109df0728e10de
I20260929 03:22:36.996474 32145 tablet_service.cc:1514] Processing DeleteTablet for tablet a0947ceda19648d093109df0728e10de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:36 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:33910
I20260929 03:22:36.996555 32411 tablet_service.cc:1514] Processing DeleteTablet for tablet a0947ceda19648d093109df0728e10de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:36 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:37966
I20260929 03:22:36.996568 32278 tablet_service.cc:1514] Processing DeleteTablet for tablet a0947ceda19648d093109df0728e10de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:36 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:38086
I20260929 03:22:36.996925 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32083
I20260929 03:22:36.996865 32518 tablet_replica.cc:330] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: stopping tablet replica
I20260929 03:22:36.997701 32518 raft_consensus.cc:2229] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:36.997828 32518 raft_consensus.cc:2258] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:36.997994 32519 tablet_replica.cc:330] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: stopping tablet replica
I20260929 03:22:36.998097 32519 raft_consensus.cc:2229] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:36.998198 32519 raft_consensus.cc:2258] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:36.998607 32519 ts_tablet_manager.cc:1905] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:36.998618 32518 ts_tablet_manager.cc:1905] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:37.000169 32519 ts_tablet_manager.cc:1918] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:37.000224 32518 ts_tablet_manager.cc:1918] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:37.000264 32519 log.cc:1196] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/a0947ceda19648d093109df0728e10de
I20260929 03:22:37.000285 32518 log.cc:1196] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/a0947ceda19648d093109df0728e10de
I20260929 03:22:37.000551 32519 ts_tablet_manager.cc:1939] T a0947ceda19648d093109df0728e10de P d5118ae04d924027811f1fc6bac00b31: Deleting consensus metadata
I20260929 03:22:37.000556 32518 ts_tablet_manager.cc:1939] T a0947ceda19648d093109df0728e10de P c84ae2c896b04310a19103b74a409f60: Deleting consensus metadata
I20260929 03:22:37.001003 32008 catalog_manager.cc:4841] TS c84ae2c896b04310a19103b74a409f60 (127.27.68.67:34293): tablet a0947ceda19648d093109df0728e10de (table test-table [id=8b168cacc0744674b6e23187a25698ba]) successfully deleted
I20260929 03:22:37.001114 32008 catalog_manager.cc:4841] TS d5118ae04d924027811f1fc6bac00b31 (127.27.68.66:46617): tablet a0947ceda19648d093109df0728e10de (table test-table [id=8b168cacc0744674b6e23187a25698ba]) successfully deleted
W20260929 03:22:37.003295 32007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:38753 (error 108)
I20260929 03:22:37.003878 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32216
I20260929 03:22:37.003893 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.003741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38753 (local address 127.0.0.1:33922)
0929 03:22:37.003810 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.003854 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 03:22:37.007735 32007 catalog_manager.cc:4584] TS 2e4bacc3d3a5499fa3f04edac553a333 (127.27.68.65:38753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0947ceda19648d093109df0728e10de: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38753: connect: Connection refused (error 111)
I20260929 03:22:37.008985 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32349
I20260929 03:22:37.013648 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 31990
2026-09-29T03:22:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1994 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:37Z Disabled control of system clock
I20260929 03:22:37.041735 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33715
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:33715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:22:37.122933 32524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.123152 32524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.123174 32524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.124392 32524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:37.124441 32524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.124454 32524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.124467 32524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:37.124480 32524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:37.125802 32524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:33715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.126015 32524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.126269 32524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:37.131179 32529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.131179 32530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.131286 32532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.131721 32524 server_base.cc:1034] running on GCE node
I20260929 03:22:37.131937 32524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:37.132254 32524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:37.133405 32524 hybrid_clock.cc:648] HybridClock initialized: now 1790652157133387 us; error 34 us; skew 500 ppm
I20260929 03:22:37.139896 32524 webserver.cc:460] Webserver started at http://127.27.68.126:46829/ using document root <none> and password file <none>
I20260929 03:22:37.140237 32524 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:37.140297 32524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:37.140504 32524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:37.141844 32524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "06c7d8906bbe445aaf5fa4b03a1884b3"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "280cce2dce5a95478d068f69183ac393"
server_key_iv: "0bd837d6a166289f59b12150c4338da6"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.142261 32524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "06c7d8906bbe445aaf5fa4b03a1884b3"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "280cce2dce5a95478d068f69183ac393"
server_key_iv: "0bd837d6a166289f59b12150c4338da6"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.143847 32524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260929 03:22:37.144709 32538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.144914 32524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:37.145036 32524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "06c7d8906bbe445aaf5fa4b03a1884b3"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "280cce2dce5a95478d068f69183ac393"
server_key_iv: "0bd837d6a166289f59b12150c4338da6"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.145126 32524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:37.148859 32524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:37.149168 32524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:37.149339 32524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:37.153815 32524 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:33715
I20260929 03:22:37.153860 32590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:33715 every 8 connection(s)
I20260929 03:22:37.154271 32524 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:37.154982 32591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:37.156183 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32524
I20260929 03:22:37.156283 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:37.156471 27921 external_mini_cluster.cc:1428] Setting key 0226e407e470bf6da72ca5433210e9b9
I20260929 03:22:37.158660 32591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: Bootstrap starting.
I20260929 03:22:37.158993 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.157084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.27.68.126:33715)
0929 03:22:37.157282 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.157290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.157316 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:22:37.157717 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.157722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.157761 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.157876 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.158550 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.158553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.158554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.158557 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.158595 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.158606 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.158659 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.158781 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.158847 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260929 03:22:37.159438 32591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:37.159754 32591 log.cc:824] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:37.160450 32591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: No bootstrap required, opened a new log
I20260929 03:22:37.161777 32591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } }
I20260929 03:22:37.161914 32591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:37.161936 32591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c7d8906bbe445aaf5fa4b03a1884b3, State: Initialized, Role: FOLLOWER
I20260929 03:22:37.162062 32591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } }
I20260929 03:22:37.162151 32591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:37.162181 32591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:37.162232 32591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:37.162837 32591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } }
I20260929 03:22:37.162922 32591 leader_election.cc:302] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c7d8906bbe445aaf5fa4b03a1884b3; no voters:
I20260929 03:22:37.163107 32591 leader_election.cc:288] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:37.163399 32591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:37.163584 32596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:37.163744 32596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [term 1 LEADER]: Becoming Leader. State: Replica: 06c7d8906bbe445aaf5fa4b03a1884b3, State: Running, Role: LEADER
I20260929 03:22:37.163879 32596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } }
I20260929 03:22:37.164369 32598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c7d8906bbe445aaf5fa4b03a1884b3. Latest consensus state: current_term: 1 leader_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } } }
I20260929 03:22:37.164461 32598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:37.164525 32597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c7d8906bbe445aaf5fa4b03a1884b3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 33715 } } }
I20260929 03:22:37.164623 32597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:37.164670 32609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:37.165478 32609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:37.166929 32609 catalog_manager.cc:1347] Generated new cluster ID: e0902d505e424d4b9483bbd80d45fc8b
I20260929 03:22:37.166985 32609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:37.172247 32609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:37.172443 32609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:37.177706 32609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: Generated new TSK 0
I20260929 03:22:37.177911 32609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:22:37.186859 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--builtin_ntp_servers=127.27.68.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:22:37.276160 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.276382 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.276405 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.277719 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.277767 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.277796 32615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:37.279199 32615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.279481 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.279796 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:37.285588 32621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.285642 32620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.285583 32623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.285960 32615 server_base.cc:1034] running on GCE node
I20260929 03:22:37.286170 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:37.286427 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:37.287585 32615 hybrid_clock.cc:648] HybridClock initialized: now 1790652157287573 us; error 32 us; skew 500 ppm
I20260929 03:22:37.288847 32615 webserver.cc:460] Webserver started at http://127.27.68.65:35467/ using document root <none> and password file <none>
I20260929 03:22:37.289111 32615 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:37.289182 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:37.289342 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:37.290697 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "cf232cfb15914808b9fc05ff0d6cd380"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "dfbb43f04c77158951af1c827c65d0b1"
server_key_iv: "e2d84ff96fee1877b7428e27ca81476f"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.291090 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "cf232cfb15914808b9fc05ff0d6cd380"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "dfbb43f04c77158951af1c827c65d0b1"
server_key_iv: "e2d84ff96fee1877b7428e27ca81476f"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.292587 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:37.293450 32629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.293664 32615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:37.293782 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "cf232cfb15914808b9fc05ff0d6cd380"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "dfbb43f04c77158951af1c827c65d0b1"
server_key_iv: "e2d84ff96fee1877b7428e27ca81476f"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.293871 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:37.297320 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:37.297614 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:37.297770 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:37.297994 32615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:37.298321 32615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:37.298374 32615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.298415 32615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:37.298442 32615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.304705 32615 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:32811
I20260929 03:22:37.304736 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:32811 every 8 connection(s)
I20260929 03:22:37.305096 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:37.308180 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.305893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34717 (local address 127.27.68.126:33715)
0929 03:22:37.305959 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.305962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.306218 (+ 256us) server_negotiation.cc:407] Connection header received
0929 03:22:37.306689 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.306691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.306696 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.306766 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.307970 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.307971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.307972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.307976 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.307991 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.307994 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.308054 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.308111 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.308124 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:22:37.308373 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.305878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33715 (local address 127.27.68.65:34717)
0929 03:22:37.306077 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.306119 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.306588 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.306836 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.306839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.306847 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.307814 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.307815 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.307843 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.307874 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.308055 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.308057 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.308059 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.308112 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.308163 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260929 03:22:37.308861 32743 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33715
I20260929 03:22:37.308971 32743 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:37.309134 32743 heartbeater.cc:511] Master 127.27.68.126:33715 requested a full tablet report, sending...
I20260929 03:22:37.309641 32555 ts_manager.cc:194] Registered new tserver with Master: cf232cfb15914808b9fc05ff0d6cd380 (127.27.68.65:32811)
I20260929 03:22:37.312407 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32615
I20260929 03:22:37.312498 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:37.312682 27921 external_mini_cluster.cc:1428] Setting key f59169da665d3fa37b8536a8564ffa9b
I20260929 03:22:37.313566 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--builtin_ntp_servers=127.27.68.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:22:37.395051 32746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.395406 32746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.395452 32746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.396798 32746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.396863 32746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.396909 32746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:37.398463 32746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.398770 32746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.399093 32746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:37.404090 32754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.404129 32752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.404232 32746 server_base.cc:1034] running on GCE node
W20260929 03:22:37.404307 32751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.404564 32746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:37.404778 32746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:37.405923 32746 hybrid_clock.cc:648] HybridClock initialized: now 1790652157405892 us; error 45 us; skew 500 ppm
I20260929 03:22:37.407337 32746 webserver.cc:460] Webserver started at http://127.27.68.66:46469/ using document root <none> and password file <none>
I20260929 03:22:37.407610 32746 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:37.407667 32746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:37.407807 32746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:37.408871 32746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "1f86af8bf2ff27d00315aeac8b8cee71"
server_key_iv: "f981a30da7c69a91662fe7ea975579c4"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.409189 32746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "1f86af8bf2ff27d00315aeac8b8cee71"
server_key_iv: "f981a30da7c69a91662fe7ea975579c4"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.410526 32746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:37.411252 32760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.411442 32746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:37.411550 32746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "1f86af8bf2ff27d00315aeac8b8cee71"
server_key_iv: "f981a30da7c69a91662fe7ea975579c4"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.411640 32746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:37.416450 32746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:37.416755 32746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:37.416908 32746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:37.417125 32746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:37.417448 32746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:37.417510 32746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.417552 32746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:37.417580 32746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.423422 32746 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:43769
I20260929 03:22:37.423461 405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:43769 every 8 connection(s)
I20260929 03:22:37.423823 32746 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:37.426631 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.424587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:57119 (local address 127.27.68.126:33715)
0929 03:22:37.424632 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.424634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.424835 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:22:37.425311 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.425312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.425317 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.425365 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.426436 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.426438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.426438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.426442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.426455 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.426458 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.426504 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.426565 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.426577 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:37.426805 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.424554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33715 (local address 127.27.68.66:57119)
0929 03:22:37.424730 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.424764 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.425219 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.425427 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.425430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.425439 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.426299 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.426300 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.426328 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.426355 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.426507 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.426510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.426512 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.426564 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.426611 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260929 03:22:37.427305 406 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33715
I20260929 03:22:37.427397 406 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:37.427510 406 heartbeater.cc:511] Master 127.27.68.126:33715 requested a full tablet report, sending...
I20260929 03:22:37.427863 32555 ts_manager.cc:194] Registered new tserver with Master: 3a4b1f3efaed4efc8ec3f946ad2e967b (127.27.68.66:43769)
I20260929 03:22:37.428864 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 32746
I20260929 03:22:37.428969 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:37.429154 27921 external_mini_cluster.cc:1428] Setting key 35ac85a1d8d50dfa293f8486a1a6c45b
I20260929 03:22:37.429968 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--builtin_ntp_servers=127.27.68.84:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:22:37.511477 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.511688 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.511713 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.512880 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.512923 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.512948 409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:37.514257 409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.514467 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.514735 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:37.520120 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.520052 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.520057 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.520735 409 server_base.cc:1034] running on GCE node
I20260929 03:22:37.520905 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:37.521116 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:37.522256 409 hybrid_clock.cc:648] HybridClock initialized: now 1790652157522245 us; error 30 us; skew 500 ppm
I20260929 03:22:37.523667 409 webserver.cc:460] Webserver started at http://127.27.68.67:46333/ using document root <none> and password file <none>
I20260929 03:22:37.523953 409 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:37.524005 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:37.524205 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:37.525323 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "9280d2ff007e4af3b6a9ec122f53dcea"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "4f9350503934f8265e9488a43f18ee7e"
server_key_iv: "829bb3de4f990e6a73cec20272ca92ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.525697 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "9280d2ff007e4af3b6a9ec122f53dcea"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "4f9350503934f8265e9488a43f18ee7e"
server_key_iv: "829bb3de4f990e6a73cec20272ca92ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.527021 409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:22:37.527895 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.528103 409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:37.528215 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "9280d2ff007e4af3b6a9ec122f53dcea"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "4f9350503934f8265e9488a43f18ee7e"
server_key_iv: "829bb3de4f990e6a73cec20272ca92ae"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.528321 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:37.531827 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:37.532114 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:37.532274 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:37.532482 409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:37.532791 409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:37.532842 409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.532882 409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:37.532910 409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.538800 409 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:33545
I20260929 03:22:37.538841 536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:33545 every 8 connection(s)
I20260929 03:22:37.539252 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:37.542294 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.540010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:46231 (local address 127.27.68.126:33715)
0929 03:22:37.540091 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.540093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.540341 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:22:37.540855 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.540856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.540862 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.540927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.542071 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.542073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.542073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.542077 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.542092 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.542096 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.542151 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.542219 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.542233 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:22:37.542497 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.540005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:33715 (local address 127.27.68.67:46231)
0929 03:22:37.540206 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.540248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.540758 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.541006 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.541010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.541018 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.541911 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.541912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.541942 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.541971 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.542151 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.542153 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.542156 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.542212 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.542263 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260929 03:22:37.543025 537 heartbeater.cc:347] Connected to a master server at 127.27.68.126:33715
I20260929 03:22:37.543118 537 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:37.543288 537 heartbeater.cc:511] Master 127.27.68.126:33715 requested a full tablet report, sending...
I20260929 03:22:37.543627 32555 ts_manager.cc:194] Registered new tserver with Master: 9280d2ff007e4af3b6a9ec122f53dcea (127.27.68.67:33545)
I20260929 03:22:37.544554 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 409
I20260929 03:22:37.544663 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:37.544852 27921 external_mini_cluster.cc:1428] Setting key 65b97a7a131ed20c74bea28e1532c454
I20260929 03:22:37.545718 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:22:37.548172 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.547004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.27.68.126:33715)
0929 03:22:37.547074 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.547076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.547199 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:22:37.547313 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.547314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.547318 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.547364 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.547935 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.547937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.547937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.547941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.547955 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.547958 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.548003 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.548055 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.548066 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:22:37.549728 32555 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:37.551266 32555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:37.555009 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:33545 (local address 127.0.0.1:50626)
0929 03:22:37.553807 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.553818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.553882 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.554028 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.554034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.554041 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.554540 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.554541 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.554559 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.554584 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.554849 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.554856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.554860 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.554923 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.554937 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260929 03:22:37.555248 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43769 (local address 127.0.0.1:60380)
0929 03:22:37.553709 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.553754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.553813 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.554394 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.554395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.554397 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.554820 (+ 423us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.554821 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.554827 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.554837 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.555065 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.555065 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.555066 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.555112 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.555127 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":82,"threads_started":1}
I20260929 03:22:37.555271 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.27.68.67:33545)
0929 03:22:37.553779 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.553784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.553901 (+ 117us) server_negotiation.cc:407] Connection header received
0929 03:22:37.553936 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.553938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.553945 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.554002 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.554669 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.554677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.554677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.554682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.554726 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.554752 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.554820 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.555146 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.555164 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260929 03:22:37.555506 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.27.68.66:43769)
0929 03:22:37.554005 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.554008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.554018 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:37.554043 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.554045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.554055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.554128 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.554916 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.554923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.554923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.554927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.554961 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.554982 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.555039 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.555390 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.555406 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":99,"threads_started":1}
I20260929 03:22:37.556015 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:32811 (local address 127.0.0.1:33008)
0929 03:22:37.554590 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.554599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.554641 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.555235 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.555235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.555237 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.555627 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.555628 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.555633 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.555644 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.555882 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.555883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.555884 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.555943 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.555955 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":91,"threads_started":1}
I20260929 03:22:37.556036 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.553695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.27.68.65:32811)
0929 03:22:37.553875 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.553880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.554709 (+ 829us) server_negotiation.cc:407] Connection header received
0929 03:22:37.554739 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.554742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.554752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.554818 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.555734 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.555742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.555742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.555746 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.555780 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.555808 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.555869 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.555945 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.555960 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":72,"threads_started":1}
I20260929 03:22:37.556216 340 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a5d1e42334e909e28c982faead6dd (DEFAULT_TABLE table=test-table [id=15585df20e654556a8a82372de284e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:37.556216 471 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a5d1e42334e909e28c982faead6dd (DEFAULT_TABLE table=test-table [id=15585df20e654556a8a82372de284e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:37.556540 340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a5d1e42334e909e28c982faead6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:37.556598 471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a5d1e42334e909e28c982faead6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:37.556684 32677 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4a5d1e42334e909e28c982faead6dd (DEFAULT_TABLE table=test-table [id=15585df20e654556a8a82372de284e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:37.556972 32677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4a5d1e42334e909e28c982faead6dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:37.559298 551 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Bootstrap starting.
I20260929 03:22:37.559857 551 tablet_bootstrap.cc:654] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:37.559953 552 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Bootstrap starting.
I20260929 03:22:37.560245 551 log.cc:824] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:37.560597 553 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: Bootstrap starting.
I20260929 03:22:37.560737 552 tablet_bootstrap.cc:654] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:37.561055 551 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: No bootstrap required, opened a new log
I20260929 03:22:37.561041 552 log.cc:824] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:37.561124 551 ts_tablet_manager.cc:1397] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:37.561759 552 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: No bootstrap required, opened a new log
I20260929 03:22:37.561852 552 ts_tablet_manager.cc:1397] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:37.563899 553 tablet_bootstrap.cc:654] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:37.563844 551 raft_consensus.cc:348] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.564047 551 raft_consensus.cc:374] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:37.564096 551 raft_consensus.cc:729] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4b1f3efaed4efc8ec3f946ad2e967b, State: Initialized, Role: FOLLOWER
I20260929 03:22:37.564239 553 log.cc:824] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:37.564230 551 consensus_queue.cc:260] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.564499 551 ts_tablet_manager.cc:1428] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:22:37.564643 406 heartbeater.cc:503] Master 127.27.68.126:33715 was elected leader, sending a full tablet report...
I20260929 03:22:37.564631 552 raft_consensus.cc:348] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.564788 552 raft_consensus.cc:374] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:37.564827 552 raft_consensus.cc:729] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9280d2ff007e4af3b6a9ec122f53dcea, State: Initialized, Role: FOLLOWER
I20260929 03:22:37.564942 552 consensus_queue.cc:260] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.565057 553 tablet_bootstrap.cc:492] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: No bootstrap required, opened a new log
I20260929 03:22:37.565125 553 ts_tablet_manager.cc:1397] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:22:37.565208 552 ts_tablet_manager.cc:1428] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:37.565318 537 heartbeater.cc:503] Master 127.27.68.126:33715 was elected leader, sending a full tablet report...
I20260929 03:22:37.567453 553 raft_consensus.cc:348] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.567607 553 raft_consensus.cc:374] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:37.567653 553 raft_consensus.cc:729] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf232cfb15914808b9fc05ff0d6cd380, State: Initialized, Role: FOLLOWER
I20260929 03:22:37.567770 553 consensus_queue.cc:260] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.568024 553 ts_tablet_manager.cc:1428] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:37.568064 32743 heartbeater.cc:503] Master 127.27.68.126:33715 was elected leader, sending a full tablet report...
I20260929 03:22:37.607347 559 raft_consensus.cc:482] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:37.607470 559 raft_consensus.cc:504] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.607858 559 leader_election.cc:288] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a4b1f3efaed4efc8ec3f946ad2e967b (127.27.68.66:43769), 9280d2ff007e4af3b6a9ec122f53dcea (127.27.68.67:33545)
I20260929 03:22:37.609372 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.608065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:33545 (local address 127.27.68.65:50269)
0929 03:22:37.608140 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.608151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.608214 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.608438 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.608439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.608441 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.608964 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.608965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.608973 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.608987 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.609226 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.609227 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.609228 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.609273 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.609289 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 03:22:37.609407 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.608257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:50269 (local address 127.27.68.67:33545)
0929 03:22:37.608321 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.608323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.608332 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:37.608362 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.608363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.608368 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.608430 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.609085 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.609089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.609089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.609108 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.609129 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.609133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.609209 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.609288 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.609300 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:22:37.609484 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.608135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:55463 (local address 127.27.68.66:43769)
0929 03:22:37.608197 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.608199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.608414 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:22:37.608475 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.608476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.608481 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.608583 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.609249 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.609250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.609250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.609256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.609273 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.609278 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.609335 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.609395 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.609409 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:22:37.609501 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.608068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43769 (local address 127.27.68.65:55463)
0929 03:22:37.608328 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.608338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.608417 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.608590 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.608591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.608607 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:37.609145 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:22:37.609146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:37.609153 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:22:37.609165 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:37.609343 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:37.609344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:37.609345 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:37.609396 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:22:37.609408 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":56,"thread_start_us":136,"threads_started":1}
I20260929 03:22:37.609719 491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a5d1e42334e909e28c982faead6dd" candidate_uuid: "cf232cfb15914808b9fc05ff0d6cd380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" is_pre_election: true
I20260929 03:22:37.609915 491 raft_consensus.cc:2454] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf232cfb15914808b9fc05ff0d6cd380 in term 0.
I20260929 03:22:37.609927 360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a5d1e42334e909e28c982faead6dd" candidate_uuid: "cf232cfb15914808b9fc05ff0d6cd380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" is_pre_election: true
I20260929 03:22:37.610095 360 raft_consensus.cc:2454] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf232cfb15914808b9fc05ff0d6cd380 in term 0.
I20260929 03:22:37.610229 32632 leader_election.cc:302] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9280d2ff007e4af3b6a9ec122f53dcea, cf232cfb15914808b9fc05ff0d6cd380; no voters:
I20260929 03:22:37.610373 559 raft_consensus.cc:2781] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:37.610451 559 raft_consensus.cc:482] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:37.610488 559 raft_consensus.cc:3037] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:37.611286 559 raft_consensus.cc:504] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.611581 559 leader_election.cc:288] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [CANDIDATE]: Term 1 election: Requested vote from peers 3a4b1f3efaed4efc8ec3f946ad2e967b (127.27.68.66:43769), 9280d2ff007e4af3b6a9ec122f53dcea (127.27.68.67:33545)
I20260929 03:22:37.611590 360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a5d1e42334e909e28c982faead6dd" candidate_uuid: "cf232cfb15914808b9fc05ff0d6cd380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b"
I20260929 03:22:37.611642 491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4a5d1e42334e909e28c982faead6dd" candidate_uuid: "cf232cfb15914808b9fc05ff0d6cd380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9280d2ff007e4af3b6a9ec122f53dcea"
I20260929 03:22:37.611668 360 raft_consensus.cc:3037] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:37.611718 491 raft_consensus.cc:3037] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:37.612643 360 raft_consensus.cc:2454] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf232cfb15914808b9fc05ff0d6cd380 in term 1.
I20260929 03:22:37.612643 491 raft_consensus.cc:2454] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf232cfb15914808b9fc05ff0d6cd380 in term 1.
I20260929 03:22:37.612839 32632 leader_election.cc:302] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9280d2ff007e4af3b6a9ec122f53dcea, cf232cfb15914808b9fc05ff0d6cd380; no voters:
I20260929 03:22:37.612936 559 raft_consensus.cc:2781] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:37.613099 559 raft_consensus.cc:686] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [term 1 LEADER]: Becoming Leader. State: Replica: cf232cfb15914808b9fc05ff0d6cd380, State: Running, Role: LEADER
I20260929 03:22:37.613242 559 consensus_queue.cc:237] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } }
I20260929 03:22:37.614073 32555 catalog_manager.cc:5495] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf232cfb15914808b9fc05ff0d6cd380 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "cf232cfb15914808b9fc05ff0d6cd380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf232cfb15914808b9fc05ff0d6cd380" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 32811 } health_report { overall_health: HEALTHY } } }
I20260929 03:22:37.629864 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.628952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.27.68.65:32811)
0929 03:22:37.629005 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.629006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.629015 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:37.629063 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.629064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.629077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.629127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.629706 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.629707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.629707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.629710 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.629721 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.629723 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.629762 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.629802 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.629813 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:22:37.631673 360 raft_consensus.cc:1261] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 1 FOLLOWER]: Refusing update from remote peer cf232cfb15914808b9fc05ff0d6cd380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:37.631683 491 raft_consensus.cc:1261] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 1 FOLLOWER]: Refusing update from remote peer cf232cfb15914808b9fc05ff0d6cd380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:37.632025 559 consensus_queue.cc:1035] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4b1f3efaed4efc8ec3f946ad2e967b" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:37.632174 561 consensus_queue.cc:1035] T 7c4a5d1e42334e909e28c982faead6dd P cf232cfb15914808b9fc05ff0d6cd380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9280d2ff007e4af3b6a9ec122f53dcea" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:37.634768 565 mvcc.cc:205] Tried to move back new op lower bound from 7334511237656547328 to 7334511237584564224. Current Snapshot: MvccSnapshot[applied={T|T < 7334511237656547328}]
I20260929 03:22:37.635087 566 mvcc.cc:205] Tried to move back new op lower bound from 7334511237656547328 to 7334511237584564224. Current Snapshot: MvccSnapshot[applied={T|T < 7334511237656547328}]
I20260929 03:22:37.637194 32555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:37.637290 32555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:37.638018 32555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06c7d8906bbe445aaf5fa4b03a1884b3: Sending DeleteTablet for 3 replicas of tablet 7c4a5d1e42334e909e28c982faead6dd
I20260929 03:22:37.638355 471 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a5d1e42334e909e28c982faead6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:37 UTC) from {username='slave'} at 127.0.0.1:50626
I20260929 03:22:37.638423 340 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a5d1e42334e909e28c982faead6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:37 UTC) from {username='slave'} at 127.0.0.1:60380
I20260929 03:22:37.638648 575 tablet_replica.cc:330] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: stopping tablet replica
I20260929 03:22:37.638643 32677 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4a5d1e42334e909e28c982faead6dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:37 UTC) from {username='slave'} at 127.0.0.1:33008
I20260929 03:22:37.638823 575 raft_consensus.cc:2229] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:37.638865 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32615
I20260929 03:22:37.639003 574 tablet_replica.cc:330] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: stopping tablet replica
I20260929 03:22:37.639093 574 raft_consensus.cc:2229] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:37.639649 574 raft_consensus.cc:2258] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:37.640136 574 ts_tablet_manager.cc:1905] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:37.642082 574 ts_tablet_manager.cc:1918] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:37.642138 574 log.cc:1196] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/7c4a5d1e42334e909e28c982faead6dd
I20260929 03:22:37.642328 574 ts_tablet_manager.cc:1939] T 7c4a5d1e42334e909e28c982faead6dd P 9280d2ff007e4af3b6a9ec122f53dcea: Deleting consensus metadata
I20260929 03:22:37.642743 32541 catalog_manager.cc:4841] TS 9280d2ff007e4af3b6a9ec122f53dcea (127.27.68.67:33545): tablet 7c4a5d1e42334e909e28c982faead6dd (table test-table [id=15585df20e654556a8a82372de284e01]) successfully deleted
I20260929 03:22:37.643854 575 raft_consensus.cc:2258] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 03:22:37.644155 32540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:32811 (error 108)
I20260929 03:22:37.644431 575 ts_tablet_manager.cc:1905] T 7c4a5d1e42334e909e28c982faead6dd P 3a4b1f3efaed4efc8ec3f946ad2e967b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:37.644922 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.644784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:32811 (local address 127.0.0.1:33028)
0929 03:22:37.644840 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.644887 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:32811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:22:37.644939 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32746
W20260929 03:22:37.646133 32540 catalog_manager.cc:4584] TS cf232cfb15914808b9fc05ff0d6cd380 (127.27.68.65:32811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c4a5d1e42334e909e28c982faead6dd: Network error: Client connection negotiation failed: client connection to 127.27.68.65:32811: connect: Connection refused (error 111)
I20260929 03:22:37.650027 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 409
I20260929 03:22:37.650170 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.650027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43769 (local address 127.0.0.1:60384)
0929 03:22:37.650091 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.650128 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.66:43769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:22:37.654912 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 32524
2026-09-29T03:22:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (640 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:22:37.674827 27921 mini_oidc.cc:152] Starting JWKS server
I20260929 03:22:37.676349 27921 webserver.cc:460] Webserver started at https://127.0.0.1:43091/ using document root <none> and password file <none>
I20260929 03:22:37.676479 27921 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:22:37.676669 27921 webserver.cc:460] Webserver started at http://127.0.0.1:39299/ using document root <none> and password file <none>
2026-09-29T03:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:37Z Disabled control of system clock
I20260929 03:22:37.684702 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42675
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/testchainca.cert with env {}
W20260929 03:22:37.765718 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.765923 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.765944 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.767148 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:37.767195 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.767208 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.767239 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:37.767251 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:22:37.767263 583 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:22:37.767275 583 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43091/jwks.json
I20260929 03:22:37.768947 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.769213 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.769443 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:37.774459 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:37.774473 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.774672 583 server_base.cc:1034] running on GCE node
W20260929 03:22:37.774470 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:37.775005 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:37.775385 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:37.776561 583 hybrid_clock.cc:648] HybridClock initialized: now 1790652157776554 us; error 37 us; skew 500 ppm
I20260929 03:22:37.777820 583 webserver.cc:460] Webserver started at http://127.27.68.126:34509/ using document root <none> and password file <none>
I20260929 03:22:37.778081 583 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:37.778152 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:37.778297 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:37.779415 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "1d1a6dda402f484781dcd574022db8b4"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "caf42bd2d5ad1b17c2b97ed1556dafe7"
server_key_iv: "58c33c55ffb6625df5d2318b21bfcf78"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.779781 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "1d1a6dda402f484781dcd574022db8b4"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "caf42bd2d5ad1b17c2b97ed1556dafe7"
server_key_iv: "58c33c55ffb6625df5d2318b21bfcf78"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.781101 583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:37.781787 597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:37.781958 583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:37.782066 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "1d1a6dda402f484781dcd574022db8b4"
format_stamp: "Formatted at 2026-09-29 03:22:37 on dist-test-slave-fzwk"
server_key: "caf42bd2d5ad1b17c2b97ed1556dafe7"
server_key_iv: "58c33c55ffb6625df5d2318b21bfcf78"
server_key_version: "encryptionkey@0"
I20260929 03:22:37.782145 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:37.796500 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:37.796846 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:37.797005 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:37.800879 583 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42675
I20260929 03:22:37.800915 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42675 every 8 connection(s)
I20260929 03:22:37.801270 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:37.802244 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:37.804661 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4: Bootstrap starting.
I20260929 03:22:37.805279 650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:37.805590 650 log.cc:824] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:37.806273 650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4: No bootstrap required, opened a new log
I20260929 03:22:37.807888 650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } }
I20260929 03:22:37.808030 650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:37.808054 650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1a6dda402f484781dcd574022db8b4, State: Initialized, Role: FOLLOWER
I20260929 03:22:37.808238 650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } }
I20260929 03:22:37.808306 650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:37.808354 650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:37.808405 650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:37.808964 650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } }
I20260929 03:22:37.809113 650 leader_election.cc:302] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d1a6dda402f484781dcd574022db8b4; no voters:
I20260929 03:22:37.809201 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 583
I20260929 03:22:37.809293 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:37.809304 650 leader_election.cc:288] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:37.809374 653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:37.809491 27921 external_mini_cluster.cc:1428] Setting key e0de01f8ff87313de89354fb7f4785cd
I20260929 03:22:37.809549 653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1d1a6dda402f484781dcd574022db8b4, State: Running, Role: LEADER
I20260929 03:22:37.809583 650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:37.809659 653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } }
I20260929 03:22:37.810133 655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d1a6dda402f484781dcd574022db8b4. Latest consensus state: current_term: 1 leader_uuid: "1d1a6dda402f484781dcd574022db8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } } }
I20260929 03:22:37.810247 655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:37.810226 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d1a6dda402f484781dcd574022db8b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d1a6dda402f484781dcd574022db8b4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42675 } } }
I20260929 03:22:37.810832 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:37.810828 664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:37.811599 664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:37.813087 664 catalog_manager.cc:1347] Generated new cluster ID: 016dfd0c6b8641d3bd4693e6b00ac26d
I20260929 03:22:37.813138 664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:37.816032 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.809952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.27.68.126:42675)
0929 03:22:37.810383 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.810392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.810427 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:22:37.810550 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.810559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.810645 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.811071 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:37.811257 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.812667 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.813817 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.814464 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.814595 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:37.815306 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:37.815310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:37.815310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:37.815519 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:37.815563 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:37.815578 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:37.815655 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:37.815724 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:22:37.815808 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":70,"threads_started":1}
I20260929 03:22:37.817003 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:22:37.837575 664 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:37.838146 664 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:37.841486 664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d1a6dda402f484781dcd574022db8b4: Generated new TSK 0
I20260929 03:22:37.841706 664 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:22:37.868139 680 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:22:37.868355 680 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:22:37.868568 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:37.852782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.27.68.126:42675)
0929 03:22:37.852836 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:37.852838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:37.852965 (+ 127us) server_negotiation.cc:407] Connection header received
0929 03:22:37.853045 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:37.853046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:37.853060 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:37.853145 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:22:37.853251 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.854025 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.854816 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.855145 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.855244 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:37.855485 (+ 241us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:22:37.868333 (+ 12848us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:22:37.868494 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 03:22:37.868630 658 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:22:37.869015 679 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:37.852689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42675 (local address 127.0.0.1:60826)
0929 03:22:37.852897 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:37.852907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:37.852966 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:37.853147 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:37.853147 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:37.853149 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:22:37.853208 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:37.853210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.854139 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.854139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:37.854726 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:37.854728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:37.855254 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:37.855255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:37.855342 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:37.855365 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:22:37.868549 (+ 13184us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:22:37.868711 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.68.126:42675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.68.126:42675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:22:37.869549 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 583
2026-09-29T03:22:37Z chronyd exiting
I20260929 03:22:37.905566 27921 test_util.cc:175] -----------------------------------------------
I20260929 03:22:37.905668 27921 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652132259900-27921-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:22:37.906436 27921 mini_oidc.cc:152] Starting JWKS server
I20260929 03:22:37.907173 27921 webserver.cc:460] Webserver started at https://127.0.0.1:33225/ using document root <none> and password file <none>
I20260929 03:22:37.907263 27921 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:22:37.907472 27921 webserver.cc:460] Webserver started at http://127.0.0.1:33869/ using document root <none> and password file <none>
2026-09-29T03:22:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:37Z Disabled control of system clock
I20260929 03:22:37.915093 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35677
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35677
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33225/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/testchainca.cert with env {}
W20260929 03:22:37.996122 686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:37.996352 686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:37.996379 686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:37.997648 686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:37.997697 686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:37.997710 686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:37.997723 686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:37.997780 686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:22:37.997799 686 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:22:37.997822 686 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33225/jwks.json
I20260929 03:22:37.999464 686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33225/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:37.999747 686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:37.999982 686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:38.004885 692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.004971 691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.004900 694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:38.005116 686 server_base.cc:1034] running on GCE node
I20260929 03:22:38.005344 686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:38.005602 686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:38.006754 686 hybrid_clock.cc:648] HybridClock initialized: now 1790652158006740 us; error 28 us; skew 500 ppm
I20260929 03:22:38.008103 686 webserver.cc:460] Webserver started at http://127.27.68.126:41349/ using document root <none> and password file <none>
I20260929 03:22:38.008361 686 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:38.008409 686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:38.008572 686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:38.009701 686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "2a84b8be0107401f8f3a5440f8c0322f"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "e668e675ed29809cd36d08699b9fe926"
server_key_iv: "4b7f7e57508abae7343629eb6368111c"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.010097 686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "2a84b8be0107401f8f3a5440f8c0322f"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "e668e675ed29809cd36d08699b9fe926"
server_key_iv: "4b7f7e57508abae7343629eb6368111c"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.011570 686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:38.012408 700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:38.012583 686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:38.012694 686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "2a84b8be0107401f8f3a5440f8c0322f"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "e668e675ed29809cd36d08699b9fe926"
server_key_iv: "4b7f7e57508abae7343629eb6368111c"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.012776 686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:38.028616 686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:38.028985 686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:38.029157 686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:38.033387 686 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35677
I20260929 03:22:38.033417 752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35677 every 8 connection(s)
I20260929 03:22:38.033788 686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:38.034433 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:38.037248 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f: Bootstrap starting.
I20260929 03:22:38.037849 753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:38.038136 753 log.cc:824] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:38.038842 753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f: No bootstrap required, opened a new log
I20260929 03:22:38.040169 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 686
I20260929 03:22:38.040256 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:38.040413 27921 external_mini_cluster.cc:1428] Setting key cc42cc5fc703aab6f9472243b1b5c30c
I20260929 03:22:38.040673 753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } }
I20260929 03:22:38.040864 753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:38.040902 753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a84b8be0107401f8f3a5440f8c0322f, State: Initialized, Role: FOLLOWER
I20260929 03:22:38.041059 753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } }
I20260929 03:22:38.041148 753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:38.041198 753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:38.041260 753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:38.042032 753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } }
I20260929 03:22:38.042155 753 leader_election.cc:302] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a84b8be0107401f8f3a5440f8c0322f; no voters:
I20260929 03:22:38.042337 753 leader_election.cc:288] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:38.042364 758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:38.042537 758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [term 1 LEADER]: Becoming Leader. State: Replica: 2a84b8be0107401f8f3a5440f8c0322f, State: Running, Role: LEADER
I20260929 03:22:38.042606 753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:38.042624 758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } }
I20260929 03:22:38.043093 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a84b8be0107401f8f3a5440f8c0322f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } } }
I20260929 03:22:38.043087 758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a84b8be0107401f8f3a5440f8c0322f. Latest consensus state: current_term: 1 leader_uuid: "2a84b8be0107401f8f3a5440f8c0322f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a84b8be0107401f8f3a5440f8c0322f" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35677 } } }
I20260929 03:22:38.043255 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:38.043269 758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:38.043635 768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:38.044149 768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:38.045804 768 catalog_manager.cc:1347] Generated new cluster ID: 154b3e2c27bc45b7854cf5932dba46f3
I20260929 03:22:38.045855 768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:38.047757 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.041158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.27.68.126:35677)
0929 03:22:38.041485 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.041494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.041877 (+ 383us) server_negotiation.cc:407] Connection header received
0929 03:22:38.041997 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.042002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.042065 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.042210 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:38.042476 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.044421 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.045560 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.046189 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.046303 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.047007 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:38.047011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:38.047012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:38.047248 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:38.047290 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:38.047304 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:38.047369 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:38.047466 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:22:38.047545 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260929 03:22:38.048918 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:22:38.058108 768 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:38.058645 768 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:38.062179 768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a84b8be0107401f8f3a5440f8c0322f: Generated new TSK 0
I20260929 03:22:38.062394 768 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:22:38.082108 784 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:22:38.082209 784 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:22:38.082579 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.068043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.27.68.126:35677)
0929 03:22:38.068114 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.068117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.068234 (+ 117us) server_negotiation.cc:407] Connection header received
0929 03:22:38.068312 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.068313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.068323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.068384 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:22:38.068551 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.069043 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.070201 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.070398 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.070450 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.070648 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:22:38.082379 (+ 11731us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:22:38.082517 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 03:22:38.082656 757 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 03:22:38.082780 783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:38.067971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35677 (local address 127.0.0.1:33322)
0929 03:22:38.068159 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:38.068171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:38.068240 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:38.068418 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:38.068419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:38.068421 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:22:38.068482 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.068484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.069130 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.069131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.070110 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.070114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.070480 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.070481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.070546 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.070552 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:22:38.082510 (+ 11958us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:22:38.082667 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.68.126:35677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33225/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260929 03:22:38.083313 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 686
2026-09-29T03:22:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:22:38.120661 27921 mini_oidc.cc:152] Starting JWKS server
I20260929 03:22:38.121387 27921 webserver.cc:460] Webserver started at https://127.0.0.1:45615/ using document root <none> and password file <none>
I20260929 03:22:38.121461 27921 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:22:38.121681 27921 webserver.cc:460] Webserver started at http://127.0.0.1:46053/ using document root <none> and password file <none>
2026-09-29T03:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:38Z Disabled control of system clock
I20260929 03:22:38.129464 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43169
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:35185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:43169
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45615/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:22:38.210649 790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:38.210865 790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:38.210886 790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:38.212260 790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:38.212316 790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:38.212329 790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:38.212342 790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:38.212354 790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:22:38.212368 790 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:22:38.212378 790 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45615/jwks.json
I20260929 03:22:38.213842 790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:43169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45615/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:38.214047 790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:38.214295 790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:38.219153 798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.219153 795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.219203 796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:38.219446 790 server_base.cc:1034] running on GCE node
I20260929 03:22:38.219745 790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:38.220057 790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:38.221238 790 hybrid_clock.cc:648] HybridClock initialized: now 1790652158221228 us; error 36 us; skew 500 ppm
I20260929 03:22:38.222680 790 webserver.cc:460] Webserver started at http://127.27.68.126:33065/ using document root <none> and password file <none>
I20260929 03:22:38.222952 790 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:38.223027 790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:38.223181 790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:38.224308 790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "935dd79c7acc400ebcc2628e411268ca"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "b859c6accbdcbb1a28936aff487d6104"
server_key_iv: "b3eccd656d6375f9c812ebda2c4379b6"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.224690 790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "935dd79c7acc400ebcc2628e411268ca"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "b859c6accbdcbb1a28936aff487d6104"
server_key_iv: "b3eccd656d6375f9c812ebda2c4379b6"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.226058 790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:38.226811 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:38.226982 790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:38.227094 790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "935dd79c7acc400ebcc2628e411268ca"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "b859c6accbdcbb1a28936aff487d6104"
server_key_iv: "b3eccd656d6375f9c812ebda2c4379b6"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.227181 790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:38.237609 790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:38.237957 790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:38.238121 790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:38.242167 790 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:43169
I20260929 03:22:38.242210 857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:43169 every 8 connection(s)
I20260929 03:22:38.242609 790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:38.243327 858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:38.243901 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 790
I20260929 03:22:38.244014 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:38.244228 27921 external_mini_cluster.cc:1428] Setting key 9273ec86e1f6913002b940d562574b2e
I20260929 03:22:38.247072 858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: Bootstrap starting.
I20260929 03:22:38.247838 858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:38.248126 858 log.cc:824] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:38.248905 858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: No bootstrap required, opened a new log
I20260929 03:22:38.250133 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.244791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.27.68.126:43169)
0929 03:22:38.244993 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.245000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.245026 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:22:38.245464 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.245471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.245538 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.245658 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:38.245909 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.246796 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.247958 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.248611 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.248753 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.249491 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:38.249496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:38.249496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:38.249673 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:38.249706 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:38.249717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:38.249793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:38.249870 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:22:38.249954 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260929 03:22:38.250994 858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } }
I20260929 03:22:38.251135 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:22:38.251183 858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:38.251255 858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935dd79c7acc400ebcc2628e411268ca, State: Initialized, Role: FOLLOWER
I20260929 03:22:38.251380 858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } }
I20260929 03:22:38.251461 858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:38.251504 858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:38.251551 858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:38.252283 858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } }
I20260929 03:22:38.252395 858 leader_election.cc:302] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 935dd79c7acc400ebcc2628e411268ca; no voters:
I20260929 03:22:38.252584 858 leader_election.cc:288] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:38.252622 863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:38.252816 863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [term 1 LEADER]: Becoming Leader. State: Replica: 935dd79c7acc400ebcc2628e411268ca, State: Running, Role: LEADER
I20260929 03:22:38.252841 858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:38.252905 863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } }
I20260929 03:22:38.253264 864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "935dd79c7acc400ebcc2628e411268ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } } }
I20260929 03:22:38.253268 865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 935dd79c7acc400ebcc2628e411268ca. Latest consensus state: current_term: 1 leader_uuid: "935dd79c7acc400ebcc2628e411268ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "935dd79c7acc400ebcc2628e411268ca" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43169 } } }
I20260929 03:22:38.253381 864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:38.253389 865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca [sys.catalog]: This master's current role is: LEADER
W20260929 03:22:38.254693 879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:22:38.254757 879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:22:38.254835 871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:38.255023 871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:38.256484 871 catalog_manager.cc:1347] Generated new cluster ID: 29dbd41c0ccf4b5cb2a45548b73de3b3
I20260929 03:22:38.256563 871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:38.283773 871 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:38.284334 871 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:38.290017 871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 935dd79c7acc400ebcc2628e411268ca: Generated new TSK 0
I20260929 03:22:38.290252 871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:22:38.301115 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.297288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.27.68.126:43169)
0929 03:22:38.297341 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.297343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.297435 (+ 92us) server_negotiation.cc:407] Connection header received
0929 03:22:38.297485 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.297486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.297532 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.297620 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:22:38.297747 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.298562 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.299426 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.299702 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.299772 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.299911 (+ 139us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:22:38.300889 (+ 978us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:22:38.301028 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260929 03:22:38.301182 861 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:22:38.302944 888 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:38.297216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43169 (local address 127.0.0.1:48666)
0929 03:22:38.297354 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:38.297365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:38.297429 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:38.297623 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:38.297624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:38.297626 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:22:38.297678 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.297679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.298679 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.298680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.299342 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.299345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.299781 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.299782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.299821 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.299826 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:22:38.300987 (+ 1161us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:22:38.302845 (+ 1858us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.68.126:43169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1658,"thread_start_us":88,"threads_started":1}
I20260929 03:22:38.303335 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 790
E20260929 03:22:38.312101 889 webserver.cc:533] Webserver: sslize failed:
E20260929 03:22:38.312171 889 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T03:22:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (220 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:38Z Disabled control of system clock
I20260929 03:22:38.348302 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34903
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:42387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34903
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/testchainca.cert with env {}
W20260929 03:22:38.431854 893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:38.432150 893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:38.432178 893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:38.433493 893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:38.433552 893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:38.433564 893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:38.433578 893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:38.433590 893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:22:38.433602 893 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:22:38.433614 893 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:22:38.435343 893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:38.435609 893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:38.435854 893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:38.440724 898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.440721 899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:38.440718 901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:38.441071 893 server_base.cc:1034] running on GCE node
I20260929 03:22:38.441342 893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:38.441632 893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:38.442782 893 hybrid_clock.cc:648] HybridClock initialized: now 1790652158442763 us; error 29 us; skew 500 ppm
I20260929 03:22:38.444321 893 webserver.cc:460] Webserver started at http://127.27.68.126:37203/ using document root <none> and password file <none>
I20260929 03:22:38.444592 893 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:38.444648 893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:38.444839 893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:38.445955 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0b5c4353492c49859c91a706ef8039b8"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "72dbbd11060f64630bf8c47af8f3d8e8"
server_key_iv: "70dfb123e8f81c9e6b278df5cef39273"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.446383 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0b5c4353492c49859c91a706ef8039b8"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "72dbbd11060f64630bf8c47af8f3d8e8"
server_key_iv: "70dfb123e8f81c9e6b278df5cef39273"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.447871 893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:22:38.448638 907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:38.448812 893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:38.448930 893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0b5c4353492c49859c91a706ef8039b8"
format_stamp: "Formatted at 2026-09-29 03:22:38 on dist-test-slave-fzwk"
server_key: "72dbbd11060f64630bf8c47af8f3d8e8"
server_key_iv: "70dfb123e8f81c9e6b278df5cef39273"
server_key_version: "encryptionkey@0"
I20260929 03:22:38.449018 893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:38.459389 893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:38.459760 893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:38.459944 893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:38.464313 893 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:34903
I20260929 03:22:38.464358 959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:34903 every 8 connection(s)
I20260929 03:22:38.464776 893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:38.465896 960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:38.468377 960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8: Bootstrap starting.
I20260929 03:22:38.468994 960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:38.469302 960 log.cc:824] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:38.470068 960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8: No bootstrap required, opened a new log
I20260929 03:22:38.471752 960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } }
I20260929 03:22:38.471903 960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:38.471926 960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5c4353492c49859c91a706ef8039b8, State: Initialized, Role: FOLLOWER
I20260929 03:22:38.472136 960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } }
I20260929 03:22:38.472235 960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:38.472290 960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:38.472342 960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:38.472683 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 893
I20260929 03:22:38.472777 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:38.472965 27921 external_mini_cluster.cc:1428] Setting key 58f1973b2c254e4921d2ee50d2d9f2c2
I20260929 03:22:38.473038 960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } }
I20260929 03:22:38.473155 960 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5c4353492c49859c91a706ef8039b8; no voters:
I20260929 03:22:38.473378 960 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:38.473410 963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:38.473569 963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5c4353492c49859c91a706ef8039b8, State: Running, Role: LEADER
I20260929 03:22:38.473657 960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:38.473662 963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } }
I20260929 03:22:38.474273 964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5c4353492c49859c91a706ef8039b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } } }
I20260929 03:22:38.474366 964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:38.474268 963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5c4353492c49859c91a706ef8039b8. Latest consensus state: current_term: 1 leader_uuid: "0b5c4353492c49859c91a706ef8039b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5c4353492c49859c91a706ef8039b8" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34903 } } }
I20260929 03:22:38.474496 963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:38.475283 974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:38.475845 974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:38.479102 974 catalog_manager.cc:1347] Generated new cluster ID: 65a2df89a4f148369bdde776e781b225
I20260929 03:22:38.479169 974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:38.480085 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.473577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.27.68.126:34903)
0929 03:22:38.474041 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.474052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.474088 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:22:38.474219 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.474226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.474310 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.474475 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:38.474866 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.476488 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.477604 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.478254 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.478408 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.479266 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:38.479271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:38.479271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:22:38.479501 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:38.479550 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:22:38.479567 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:38.479653 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:38.479740 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:22:38.479828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":69,"threads_started":1}
I20260929 03:22:38.481070 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:22:38.486019 974 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:38.486563 974 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:38.491438 974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5c4353492c49859c91a706ef8039b8: Generated new TSK 0
I20260929 03:22:38.491658 974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:22:38.580443 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:38.497499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.27.68.126:34903)
0929 03:22:38.497551 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:38.497554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:38.497712 (+ 158us) server_negotiation.cc:407] Connection header received
0929 03:22:38.497796 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:38.497797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:38.497807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:38.497870 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:22:38.498042 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.498520 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.499736 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.499928 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.499984 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.500207 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:22:38.580187 (+ 79980us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:22:38.580385 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57586: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 03:22:38.580538 967 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57586: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:22:38.580632 989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:38.497384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34903 (local address 127.0.0.1:57586)
0929 03:22:38.497615 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:38.497627 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:38.497709 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:38.497891 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:38.497892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:38.497893 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:22:38.497961 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.497963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.498626 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.498627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.499641 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:38.499645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:38.500019 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:38.500021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:38.500089 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:38.500096 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:22:38.580343 (+ 80247us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:22:38.580520 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.68.126:34903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":108,"threads_started":1}
I20260929 03:22:38.581023 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 893
2026-09-29T03:22:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (257 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:38 dist-test-slave-fzwk krb5kdc[1001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:38 dist-test-slave-fzwk krb5kdc[1001](info): set up 2 sockets
Sep 29 03:22:38 dist-test-slave-fzwk krb5kdc[1001](info): commencing operation
krb5kdc: starting...
W20260929 03:22:39.799937 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:39 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:39.921401 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41633
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:40.002827 1014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:40.003047 1014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:40.003070 1014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:40.004251 1014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:40.004294 1014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:40.004309 1014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:40.004323 1014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:40.004333 1014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:40.005828 1014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:41633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:40.006047 1014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:40.006220 1014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:40.011160 1022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.011251 1014 server_base.cc:1034] running on GCE node
W20260929 03:22:40.011160 1020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:40.011173 1019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.011708 1014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:40.012017 1014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:40.013193 1014 hybrid_clock.cc:648] HybridClock initialized: now 1790652160013178 us; error 46 us; skew 500 ppm
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.015744 1014 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:40.016144 1014 webserver.cc:460] Webserver started at http://127.27.68.126:42199/ using document root <none> and password file <none>
I20260929 03:22:40.016331 1014 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:40.016373 1014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:40.016466 1014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:40.017482 1014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "bf1023bfa9e74be2a45e0bd378bbdf75"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "a58fb7b42e06272c5d35b6c22024b160"
server_key_iv: "10361508e285a684dac81a38a33de5b5"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.017791 1014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "bf1023bfa9e74be2a45e0bd378bbdf75"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "a58fb7b42e06272c5d35b6c22024b160"
server_key_iv: "10361508e285a684dac81a38a33de5b5"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.019050 1014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:40.019814 1029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.019977 1014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.020064 1014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "bf1023bfa9e74be2a45e0bd378bbdf75"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "a58fb7b42e06272c5d35b6c22024b160"
server_key_iv: "10361508e285a684dac81a38a33de5b5"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.020150 1014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:40.039582 1014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:40.041477 1014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:40.041673 1014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:40.045655 1014 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:41633
I20260929 03:22:40.045714 1082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:41633 every 8 connection(s)
I20260929 03:22:40.046104 1014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:40.046454 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1014
I20260929 03:22:40.046568 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:40.046769 27921 external_mini_cluster.cc:1428] Setting key 8fa59d9e042c0d06771f9ce80a0e9b4a
I20260929 03:22:40.047271 1083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:40.050621 1083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75: Bootstrap starting.
I20260929 03:22:40.051334 1083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:40.051633 1083 log.cc:824] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:40.052531 1083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75: No bootstrap required, opened a new log
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:40.055037 1083 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } }
I20260929 03:22:40.055250 1083 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:40.055305 1083 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1023bfa9e74be2a45e0bd378bbdf75, State: Initialized, Role: FOLLOWER
I20260929 03:22:40.055480 1083 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } }
I20260929 03:22:40.055573 1083 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:40.055619 1083 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:40.055670 1083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:40.056425 1083 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } }
I20260929 03:22:40.056556 1083 leader_election.cc:302] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1023bfa9e74be2a45e0bd378bbdf75; no voters:
I20260929 03:22:40.056767 1083 leader_election.cc:288] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:40.056840 1088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:40.056999 1088 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [term 1 LEADER]: Becoming Leader. State: Replica: bf1023bfa9e74be2a45e0bd378bbdf75, State: Running, Role: LEADER
I20260929 03:22:40.057034 1083 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:40.057096 1088 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } }
I20260929 03:22:40.057477 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } } }
I20260929 03:22:40.057572 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:40.057813 1096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:40.058373 1096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:40.058382 1090 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1023bfa9e74be2a45e0bd378bbdf75. Latest consensus state: current_term: 1 leader_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1023bfa9e74be2a45e0bd378bbdf75" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41633 } } }
I20260929 03:22:40.058516 1090 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:40.059312 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.047393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.27.68.126:41633)
0929 03:22:40.047594 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.047603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.047638 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:22:40.048136 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.048145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.048218 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.048372 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.048969 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.050359 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.051134 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.051806 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.051917 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.054610 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.054616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.054616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.054633 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.056769 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.057257 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.057260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.057261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.057363 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.057897 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.057898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.057899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.058237 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.058374 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.058913 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.059054 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260929 03:22:40.060215 1096 catalog_manager.cc:1347] Generated new cluster ID: b5ebdace12fb4c75b4a479dd98fb7a88
I20260929 03:22:40.060273 1096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:40.083606 1096 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:40.084403 1096 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:40.096951 1096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1023bfa9e74be2a45e0bd378bbdf75: Generated new TSK 0
I20260929 03:22:40.097190 1096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:40.126163 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:40.208554 1109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:40.208779 1109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:40.208806 1109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:40.210037 1109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:40.210088 1109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:40.210119 1109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:40.211779 1109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:40.212040 1109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:40.212296 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:40.217350 1115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:40.217350 1114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.217511 1109 server_base.cc:1034] running on GCE node
W20260929 03:22:40.217381 1117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.217813 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:40.218086 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:40.219274 1109 hybrid_clock.cc:648] HybridClock initialized: now 1790652160219258 us; error 46 us; skew 500 ppm
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.221817 1109 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:40.222291 1109 webserver.cc:460] Webserver started at http://127.27.68.65:46781/ using document root <none> and password file <none>
I20260929 03:22:40.222530 1109 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:40.222604 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:40.222805 1109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:40.224012 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "a8fafb19f2ed4571bf8ffa5e2e44fd57"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "cbb85b6f9ed4d831ca7075e1e18bcd77"
server_key_iv: "4b75de8bc08d5ae83108ce3f8909410d"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.224386 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "a8fafb19f2ed4571bf8ffa5e2e44fd57"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "cbb85b6f9ed4d831ca7075e1e18bcd77"
server_key_iv: "4b75de8bc08d5ae83108ce3f8909410d"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.225759 1109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:40.226521 1124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.226701 1109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:40.226809 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "a8fafb19f2ed4571bf8ffa5e2e44fd57"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "cbb85b6f9ed4d831ca7075e1e18bcd77"
server_key_iv: "4b75de8bc08d5ae83108ce3f8909410d"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.226887 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:40.243520 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:40.245428 1109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:40.245601 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:40.245847 1109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:40.246186 1109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:40.246240 1109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.246282 1109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:40.246310 1109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.252260 1109 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:34063
I20260929 03:22:40.252310 1237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:34063 every 8 connection(s)
I20260929 03:22:40.252641 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:40.261652 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1109
I20260929 03:22:40.261767 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:40.261972 27921 external_mini_cluster.cc:1428] Setting key e1927145b4fef21be05a5fcbcba1e75d
I20260929 03:22:40.262346 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.253397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:47923 (local address 127.27.68.126:41633)
0929 03:22:40.253463 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.253464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.253663 (+ 199us) server_negotiation.cc:407] Connection header received
0929 03:22:40.254587 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.254588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.254603 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.254664 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.255304 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.255823 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.256816 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.257088 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.257173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.259481 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.259483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.259483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.259491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.260918 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.261281 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.261282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.261282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.261333 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.261549 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.261549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.261550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.261723 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.261803 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.262159 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.262254 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:40.262530 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.253371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41633 (local address 127.27.68.65:47923)
0929 03:22:40.253525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:40.253568 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:40.254490 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:40.254723 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:40.254726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:40.254990 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:40.255197 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.255201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.255912 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.255913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.256736 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.256738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.257183 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.257183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.257295 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.257766 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:40.257774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:40.259383 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:40.260990 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.260995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.260997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.261215 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.261388 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.261388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.261389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.261485 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.261849 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:40.261856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:40.261989 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 03:22:40.262116 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:22:40.262285 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260929 03:22:40.263074 1238 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41633
I20260929 03:22:40.263188 1238 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:40.263403 1238 heartbeater.cc:511] Master 127.27.68.126:41633 requested a full tablet report, sending...
I20260929 03:22:40.264011 1046 ts_manager.cc:194] Registered new tserver with Master: a8fafb19f2ed4571bf8ffa5e2e44fd57 (127.27.68.65:34063)
I20260929 03:22:40.264737 1046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:47923
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:40.287750 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:40.370767 1243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:40.371025 1243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:40.371068 1243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:40.372325 1243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:40.372407 1243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:40.372457 1243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:40.374064 1243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:40.374367 1243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:40.374634 1243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:40.379896 1248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:40.379961 1249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:40.379896 1251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.380110 1243 server_base.cc:1034] running on GCE node
I20260929 03:22:40.380347 1243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:40.380617 1243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:40.381804 1243 hybrid_clock.cc:648] HybridClock initialized: now 1790652160381780 us; error 37 us; skew 500 ppm
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.384550 1243 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:40.385094 1243 webserver.cc:460] Webserver started at http://127.27.68.66:38173/ using document root <none> and password file <none>
I20260929 03:22:40.385345 1243 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:40.385414 1243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:40.385566 1243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:40.386634 1243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "529f260b42d6435b94ca529976fa01a8"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "03c8bad22736127dd7e6545957625293"
server_key_iv: "815e22b3a2b0e760ded146ad683a7189"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.387010 1243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "529f260b42d6435b94ca529976fa01a8"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "03c8bad22736127dd7e6545957625293"
server_key_iv: "815e22b3a2b0e760ded146ad683a7189"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.388514 1243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:40.389308 1258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.389484 1243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:40.389580 1243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "529f260b42d6435b94ca529976fa01a8"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "03c8bad22736127dd7e6545957625293"
server_key_iv: "815e22b3a2b0e760ded146ad683a7189"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.389664 1243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:40.403968 1243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:40.405920 1243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:40.406095 1243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:40.406342 1243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:40.406656 1243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:40.406710 1243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.406751 1243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:40.406780 1243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.412688 1243 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:37449
I20260929 03:22:40.412724 1371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:37449 every 8 connection(s)
I20260929 03:22:40.413061 1243 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:40.422523 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.413823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:43305 (local address 127.27.68.126:41633)
0929 03:22:40.413887 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.413890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.414078 (+ 188us) server_negotiation.cc:407] Connection header received
0929 03:22:40.415070 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.415072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.415086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.415136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.415817 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.416314 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.417334 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.417527 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.417623 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.419912 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.419914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.419914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.419921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.421096 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.421507 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.421508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.421509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.421571 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.421837 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.421838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.421839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.421956 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.422029 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.422333 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.422420 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:22:40.422538 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1243
I20260929 03:22:40.422646 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:40.422780 1374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.413801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41633 (local address 127.27.68.66:43305)
0929 03:22:40.413946 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:40.413995 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:40.414951 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:40.415206 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:40.415233 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:40.415519 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:40.415719 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.415724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.416415 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.416416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.417237 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.417240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.417627 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.417628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.417763 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.418308 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:40.418317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:40.419816 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:40.421194 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.421201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.421204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.421429 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.421658 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.421660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.421660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.421769 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.422053 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:40.422058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:40.422201 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 03:22:40.422336 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 03:22:40.422499 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260929 03:22:40.422828 27921 external_mini_cluster.cc:1428] Setting key 29e290f80d1c3857fdcc7e737d4878b9
I20260929 03:22:40.423290 1372 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41633
I20260929 03:22:40.423408 1372 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:40.423592 1372 heartbeater.cc:511] Master 127.27.68.126:41633 requested a full tablet report, sending...
I20260929 03:22:40.423995 1045 ts_manager.cc:194] Registered new tserver with Master: 529f260b42d6435b94ca529976fa01a8 (127.27.68.66:37449)
I20260929 03:22:40.424508 1045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:43305
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:40.449692 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:40.535084 1377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:40.535425 1377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:40.535470 1377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:40.536705 1377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:40.536780 1377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:40.536827 1377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:40.538357 1377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:40.538642 1377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:40.538880 1377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:40.544056 1385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:40.544063 1383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.544178 1377 server_base.cc:1034] running on GCE node
W20260929 03:22:40.544253 1382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:40.544492 1377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:40.544780 1377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:40.545951 1377 hybrid_clock.cc:648] HybridClock initialized: now 1790652160545942 us; error 29 us; skew 500 ppm
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.548483 1377 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:40.548974 1377 webserver.cc:460] Webserver started at http://127.27.68.67:43937/ using document root <none> and password file <none>
I20260929 03:22:40.549214 1377 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:40.549283 1377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:40.549437 1377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:40.550519 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "35d313467c424345a9fe54762596adc2"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "f31208c76c74edc0b5232418bc4088e0"
server_key_iv: "9bb1d81c19e10c3c17555a4a9e170252"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.550894 1377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "35d313467c424345a9fe54762596adc2"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "f31208c76c74edc0b5232418bc4088e0"
server_key_iv: "9bb1d81c19e10c3c17555a4a9e170252"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.552346 1377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:40.553076 1392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.553246 1377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.553344 1377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "35d313467c424345a9fe54762596adc2"
format_stamp: "Formatted at 2026-09-29 03:22:40 on dist-test-slave-fzwk"
server_key: "f31208c76c74edc0b5232418bc4088e0"
server_key_iv: "9bb1d81c19e10c3c17555a4a9e170252"
server_key_version: "encryptionkey@0"
I20260929 03:22:40.553429 1377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:40.581369 1377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:40.583314 1377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:40.583487 1377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:40.583738 1377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:40.584076 1377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:40.584139 1377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.584175 1377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:40.584206 1377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:40.590111 1377 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:46713
I20260929 03:22:40.590145 1507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:46713 every 8 connection(s)
I20260929 03:22:40.590560 1377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:40.594702 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1377
I20260929 03:22:40.594792 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:40.594940 27921 external_mini_cluster.cc:1428] Setting key d93822ed465ec7ea9f090e32966aa2ca
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:40.600064 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.591316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:39717 (local address 127.27.68.126:41633)
0929 03:22:40.591380 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.591383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.591618 (+ 235us) server_negotiation.cc:407] Connection header received
0929 03:22:40.592699 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.592701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.592711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.592787 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.593412 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.593976 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.595252 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.595453 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.595529 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.597686 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.597688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.597689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.597696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.598848 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.599267 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.599268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.599269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.599325 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.599549 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.599550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.599550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.599653 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.599707 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.599936 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.599998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:22:40.600289 1510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.591310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41633 (local address 127.27.68.67:39717)
0929 03:22:40.591473 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:40.591517 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:40.592560 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:40.592849 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:40.592852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:40.593106 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:40.593324 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.593329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.594055 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.594056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.595126 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:40.595128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.595538 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.595539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:40.595680 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.596121 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:40.596130 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:40.597578 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:40.598919 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.598925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.598927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.599155 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.599382 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:40.599382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:40.599383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:40.599493 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:40.599707 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:40.599711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:40.599826 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 03:22:40.599909 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:40.600030 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260929 03:22:40.600762 1508 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41633
I20260929 03:22:40.600862 1508 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:40.601007 1508 heartbeater.cc:511] Master 127.27.68.126:41633 requested a full tablet report, sending...
I20260929 03:22:40.601394 1045 ts_manager.cc:194] Registered new tserver with Master: 35d313467c424345a9fe54762596adc2 (127.27.68.67:46713)
I20260929 03:22:40.601738 1045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:39717
I20260929 03:22:40.606547 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/test-admin.keytab.
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.622195 27921 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 03:22:40.623967 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:40.623159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.27.68.65:34063)
0929 03:22:40.623347 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.623352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.623363 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:40.623398 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.623400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.623414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.623499 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.623881 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42182: BlockingRecv error: recv got EOF from 127.0.0.1:42182 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
W20260929 03:22:40.624094 27921 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.625106 27921 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.625967 27921 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:40.633870 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.626482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.27.68.65:34063)
0929 03:22:40.626548 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.626550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.626607 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:22:40.626652 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.626652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.626663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.626721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.627078 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.627841 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.628534 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.628851 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.628902 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.631128 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.631144 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.631144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.631151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.632622 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.633022 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.633026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.633026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.633087 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.633293 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.633294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.633295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.633441 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.633621 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.633666 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.633752 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 03:22:40.635866 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:40.635149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.27.68.65:34063)
0929 03:22:40.635210 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.635244 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.635253 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:40.635348 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.635348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.635360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.635415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.635817 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42194: BlockingRecv error: recv got EOF from 127.0.0.1:42194 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 03:22:40.635946 27921 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.636886 27921 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.637739 27921 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:40.644548 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.638228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.27.68.65:34063)
0929 03:22:40.638275 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.638277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.638305 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:22:40.638366 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.638366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.638375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.638424 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.638754 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.639272 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.639909 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.640096 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.640145 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.642208 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.642210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.642210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.642216 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.643454 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.643807 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.643808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.643809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.643866 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.644060 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.644061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.644061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.644163 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.644264 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.644361 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.644441 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260929 03:22:40.646008 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:40.645425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.27.68.65:34063)
0929 03:22:40.645458 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.645460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.645557 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:22:40.645618 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.645619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.645628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.645682 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.645955 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42210: BlockingRecv error: recv got EOF from 127.0.0.1:42210 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260929 03:22:40.646080 27921 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652132259900-27921-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.647029 27921 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:40.647956 27921 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:22:40.654803 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:40.648472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.27.68.65:34063)
0929 03:22:40.648528 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:40.648530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:40.648593 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:22:40.648668 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:40.648669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:40.648678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:40.648748 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:40.649107 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.649577 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.650271 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:40.650466 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:40.650549 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:40.652544 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:40.652545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:40.652546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:40.652553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:40.653715 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.654043 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.654044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.654045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.654102 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:40.654301 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:40.654302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:40.654303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:40.654403 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:40.654518 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:40.654617 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:22:40.654695 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:40.655452 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1109
I20260929 03:22:40.660478 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1243
I20260929 03:22:40.665117 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1377
I20260929 03:22:40.669562 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1014
2026-09-29T03:22:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1912 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1528](info): set up 2 sockets
Sep 29 03:22:40 dist-test-slave-fzwk krb5kdc[1528](info): commencing operation
krb5kdc: starting...
W20260929 03:22:41.691735 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:41 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab.
Entry for principal oryx/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab.
I20260929 03:22:41.809909 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46757
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab
--principal=oryx/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:41.891795 1542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:41.892064 1542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:41.892107 1542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:41.893374 1542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:41.893452 1542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:41.893476 1542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:41.893498 1542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:41.893525 1542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:41.895092 1542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab
--principal=oryx/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:41.895426 1542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:41.895665 1542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:41.900593 1550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:41.900657 1548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:41.900671 1542 server_base.cc:1034] running on GCE node
W20260929 03:22:41.900593 1547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:41.901089 1542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:41.901409 1542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:41.902557 1542 hybrid_clock.cc:648] HybridClock initialized: now 1790652161902536 us; error 43 us; skew 500 ppm
Sep 29 03:22:41 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:41.905110 1542 init.cc:377] Logged in from keytab as oryx/127.27.68.126@KRBTEST.COM (short username oryx)
I20260929 03:22:41.905531 1542 webserver.cc:460] Webserver started at http://127.27.68.126:34287/ using document root <none> and password file <none>
I20260929 03:22:41.905718 1542 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:41.905759 1542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:41.905848 1542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:41.906880 1542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0ab2e724b3f546d9b8c643c36bdc5581"
format_stamp: "Formatted at 2026-09-29 03:22:41 on dist-test-slave-fzwk"
server_key: "bd23727533f80931112a5e2de0e5834b"
server_key_iv: "3fd6e4ccd0036228e55b83bb22a2fc5d"
server_key_version: "encryptionkey@0"
I20260929 03:22:41.907177 1542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0ab2e724b3f546d9b8c643c36bdc5581"
format_stamp: "Formatted at 2026-09-29 03:22:41 on dist-test-slave-fzwk"
server_key: "bd23727533f80931112a5e2de0e5834b"
server_key_iv: "3fd6e4ccd0036228e55b83bb22a2fc5d"
server_key_version: "encryptionkey@0"
I20260929 03:22:41.908543 1542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:41.909225 1557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:41.909399 1542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:41.909507 1542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0ab2e724b3f546d9b8c643c36bdc5581"
format_stamp: "Formatted at 2026-09-29 03:22:41 on dist-test-slave-fzwk"
server_key: "bd23727533f80931112a5e2de0e5834b"
server_key_iv: "3fd6e4ccd0036228e55b83bb22a2fc5d"
server_key_version: "encryptionkey@0"
I20260929 03:22:41.909585 1542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:41.921736 1542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:41.923648 1542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:41.923811 1542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:41.927747 1542 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46757
I20260929 03:22:41.927783 1609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46757 every 8 connection(s)
I20260929 03:22:41.928143 1542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:41.929152 1610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:41.931496 1610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Bootstrap starting.
I20260929 03:22:41.932067 1610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:41.932377 1610 log.cc:824] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:41.933064 1610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: No bootstrap required, opened a new log
I20260929 03:22:41.934743 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1542
I20260929 03:22:41.934849 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:41.934700 1610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } }
I20260929 03:22:41.934880 1610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:41.934927 1610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab2e724b3f546d9b8c643c36bdc5581, State: Initialized, Role: FOLLOWER
I20260929 03:22:41.935032 27921 external_mini_cluster.cc:1428] Setting key 9709585f19d2231b3b007407cacfa961
I20260929 03:22:41.935081 1610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } }
I20260929 03:22:41.935175 1610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:41.935235 1610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:41.935288 1610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:41.936388 1610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } }
I20260929 03:22:41.936515 1610 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab2e724b3f546d9b8c643c36bdc5581; no voters:
I20260929 03:22:41.936691 1610 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:41.936748 1615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:41.936906 1615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab2e724b3f546d9b8c643c36bdc5581, State: Running, Role: LEADER
I20260929 03:22:41.936947 1610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:41.937080 1615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } }
I20260929 03:22:41.937644 1617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } }
I20260929 03:22:41.937655 1616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab2e724b3f546d9b8c643c36bdc5581. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } }
I20260929 03:22:41.937811 1617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:41.937911 1616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:41.938244 1627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:41.938848 1627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:41.940562 1627 catalog_manager.cc:1347] Generated new cluster ID: f56b3d4201c04b10a17fc48e60d4415a
I20260929 03:22:41.940611 1627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:22:41 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:41.946475 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:41.935567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.27.68.126:46757)
0929 03:22:41.935731 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:41.935740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:41.935775 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:22:41.935876 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:41.935881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:41.935922 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:41.936028 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:41.936837 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:41.939099 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:41.940177 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:41.940565 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:41.940698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:41.943074 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:41.943079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:41.943079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:41.943098 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:41.944914 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:41.945290 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:41.945295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:41.945295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:41.945414 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:41.945633 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:41.945634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:41.945635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:41.945890 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:41.946057 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:41.946154 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:22:41.946301 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260929 03:22:41.968458 1627 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:41.969225 1627 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:41.981289 1627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Generated new TSK 0
I20260929 03:22:41.981534 1627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
I20260929 03:22:42.014389 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:42.109342 1636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:42.109634 1636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:42.109671 1636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:42.111660 1636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:42.111729 1636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:42.111776 1636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:42.114231 1636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:42.114543 1636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:42.114841 1636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.121183 1641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.121206 1642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.121212 1644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.121460 1636 server_base.cc:1034] running on GCE node
I20260929 03:22:42.121675 1636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.121996 1636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.123152 1636 hybrid_clock.cc:648] HybridClock initialized: now 1790652162123120 us; error 49 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.126313 1636 init.cc:377] Logged in from keytab as oryx/127.27.68.65@KRBTEST.COM (short username oryx)
I20260929 03:22:42.126842 1636 webserver.cc:460] Webserver started at http://127.27.68.65:37701/ using document root <none> and password file <none>
I20260929 03:22:42.127105 1636 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.127172 1636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.127342 1636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:42.128433 1636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "5ece50a719974528abcafc4a287fe9c2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "644db21c83a74aa5f206fe7521f62471"
server_key_iv: "28825d4e9aeb3f95eb8dcbd21044ec27"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.128816 1636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "5ece50a719974528abcafc4a287fe9c2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "644db21c83a74aa5f206fe7521f62471"
server_key_iv: "28825d4e9aeb3f95eb8dcbd21044ec27"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.130316 1636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:42.131089 1651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.131325 1636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:22:42.131435 1636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "5ece50a719974528abcafc4a287fe9c2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "644db21c83a74aa5f206fe7521f62471"
server_key_iv: "28825d4e9aeb3f95eb8dcbd21044ec27"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.131539 1636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.175871 1636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.178058 1636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:42.178260 1636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.178527 1636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:42.178879 1636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:42.178938 1636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.178984 1636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:42.179008 1636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.185160 1636 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:37789
I20260929 03:22:42.185200 1764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:37789 every 8 connection(s)
I20260929 03:22:42.185552 1636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:42.190596 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1636
I20260929 03:22:42.190703 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:42.190917 27921 external_mini_cluster.cc:1428] Setting key 4e679836a98d608fd82cd45f0bdc0e5b
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:42.195691 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.186271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:37901 (local address 127.27.68.126:46757)
0929 03:22:42.186320 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.186322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.186575 (+ 253us) server_negotiation.cc:407] Connection header received
0929 03:22:42.187126 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.187127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.187139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.187196 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.188240 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.188756 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.189795 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.190007 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.190087 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.192527 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.192528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.192529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.192537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.194200 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.194618 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.194619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.194620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.194681 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.194952 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.194953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.194954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.195082 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.195154 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.195485 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.195592 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:42.195899 1767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.186259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:37901)
0929 03:22:42.186446 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.186487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.187019 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.187308 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.187312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.187913 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.188122 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.188126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.188861 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.188862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.189690 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.189692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.190095 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.190096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.190230 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.190808 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.190818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.192422 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.194300 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.194308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.194310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.194541 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.194759 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.194760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.194761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.194878 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.195156 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.195164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.195348 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.195461 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.195633 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260929 03:22:42.196400 1765 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:42.196524 1765 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:42.196755 1765 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:42.197293 1574 ts_manager.cc:194] Registered new tserver with Master: 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:42.197906 1574 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:37901
WARNING: no policy specified for oryx/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
I20260929 03:22:42.218051 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:42.301081 1770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:42.301328 1770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:42.301353 1770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:42.302578 1770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:42.302629 1770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:42.302697 1770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:42.304244 1770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:42.304533 1770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:42.304790 1770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.309815 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.309815 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.309829 1778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.309993 1770 server_base.cc:1034] running on GCE node
I20260929 03:22:42.310269 1770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.310525 1770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.311691 1770 hybrid_clock.cc:648] HybridClock initialized: now 1790652162311674 us; error 33 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.314500 1770 init.cc:377] Logged in from keytab as oryx/127.27.68.66@KRBTEST.COM (short username oryx)
I20260929 03:22:42.315001 1770 webserver.cc:460] Webserver started at http://127.27.68.66:33083/ using document root <none> and password file <none>
I20260929 03:22:42.315305 1770 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.315387 1770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.315548 1770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:42.316679 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "73d33d0f582f4413958e5fb2dcc6428e"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "c980b7aaac6d0e08d119718c6c935b80"
server_key_iv: "bd9b53fa77dad478b09521217aa8d1be"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.317104 1770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "73d33d0f582f4413958e5fb2dcc6428e"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "c980b7aaac6d0e08d119718c6c935b80"
server_key_iv: "bd9b53fa77dad478b09521217aa8d1be"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.318480 1770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:42.319202 1785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.319419 1770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:42.319519 1770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "73d33d0f582f4413958e5fb2dcc6428e"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "c980b7aaac6d0e08d119718c6c935b80"
server_key_iv: "bd9b53fa77dad478b09521217aa8d1be"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.319609 1770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.331127 1770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.333686 1770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:42.333861 1770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.334110 1770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:42.334486 1770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:42.334542 1770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.334585 1770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:42.334627 1770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.341004 1770 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42355
I20260929 03:22:42.341043 1898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42355 every 8 connection(s)
I20260929 03:22:42.341403 1770 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:42.343135 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1770
I20260929 03:22:42.343281 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:42.343484 27921 external_mini_cluster.cc:1428] Setting key e3aa9d8086472422fb335ba646b971aa
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.68.67@KRBTEST.COM; defaulting to no policy
I20260929 03:22:42.352303 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.342201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:43693 (local address 127.27.68.126:46757)
0929 03:22:42.342271 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.342273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.342522 (+ 249us) server_negotiation.cc:407] Connection header received
0929 03:22:42.343088 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.343090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.343103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.343152 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.344447 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.344951 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.346433 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.346661 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.346741 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.349289 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.349291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.349291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.349299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.350781 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.351253 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.351254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.351255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.351317 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.351595 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.351596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.351597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.351727 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.351806 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.352087 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.352195 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:22:42.352610 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.342169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:43693)
0929 03:22:42.342392 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.342432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.342970 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.343258 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.343263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.344070 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.344352 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.344359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.345050 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.345051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.346357 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.346361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.346749 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.346750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.346888 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.347496 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.347506 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.349183 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.350892 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.350900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.350903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.351137 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.351386 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.351387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.351388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.351513 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.351796 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.351803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.351954 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.352068 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.352251 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260929 03:22:42.353119 1899 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:42.353241 1899 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:42.353457 1899 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:42.353902 1574 ts_manager.cc:194] Registered new tserver with Master: 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.67@KRBTEST.COM" created.
I20260929 03:22:42.354383 1574 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:43693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
I20260929 03:22:42.369614 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:42.453794 1904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:42.454061 1904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:42.454098 1904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:42.455381 1904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:42.455447 1904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:42.455492 1904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:42.457042 1904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:42.457342 1904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:42.457578 1904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.462738 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.462718 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.462752 1904 server_base.cc:1034] running on GCE node
W20260929 03:22:42.462699 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.463095 1904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.463397 1904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.464573 1904 hybrid_clock.cc:648] HybridClock initialized: now 1790652162464559 us; error 29 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.467159 1904 init.cc:377] Logged in from keytab as oryx/127.27.68.67@KRBTEST.COM (short username oryx)
I20260929 03:22:42.467703 1904 webserver.cc:460] Webserver started at http://127.27.68.67:46199/ using document root <none> and password file <none>
I20260929 03:22:42.467932 1904 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.467978 1904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.468076 1904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:42.469126 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "f82b909bbac847c5bbca02862f1e3bf2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "68dd5d5a130b34bb57cf43e403886e95"
server_key_iv: "e2cc2b3135946ca0fa3ec027218a1c67"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.469436 1904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "f82b909bbac847c5bbca02862f1e3bf2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "68dd5d5a130b34bb57cf43e403886e95"
server_key_iv: "e2cc2b3135946ca0fa3ec027218a1c67"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.470767 1904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:22:42.471541 1919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.471738 1904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:42.471838 1904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "f82b909bbac847c5bbca02862f1e3bf2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "68dd5d5a130b34bb57cf43e403886e95"
server_key_iv: "e2cc2b3135946ca0fa3ec027218a1c67"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.471920 1904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.486745 1904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.488771 1904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:42.488941 1904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.489180 1904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:42.489523 1904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:42.489557 1904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.489599 1904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:42.489630 1904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.495787 1904 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:38875
I20260929 03:22:42.495826 2032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:38875 every 8 connection(s)
I20260929 03:22:42.496202 1904 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:42.504886 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 1904
I20260929 03:22:42.504997 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:42.505201 27921 external_mini_cluster.cc:1428] Setting key 42f7777039211e917de569ce29a244bf
I20260929 03:22:42.505461 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.496943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:42505 (local address 127.27.68.126:46757)
0929 03:22:42.497028 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.497031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.497254 (+ 223us) server_negotiation.cc:407] Connection header received
0929 03:22:42.497748 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.497749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.497759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.497812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.498759 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.499341 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.500310 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.500494 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.500565 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.502802 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.502804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.502804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.502812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.504090 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.504462 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.504463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.504463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.504512 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.504720 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.504721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.504721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.504827 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.504914 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.505275 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.505359 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:22:42.505667 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.496916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:42505)
0929 03:22:42.497092 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.497129 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.497658 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.497870 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.497873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.498478 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.498676 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.498682 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.499417 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.499417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.500233 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.500235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.500589 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.500590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.500694 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.501258 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.501267 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.502711 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.504162 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.504168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.504170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.504395 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.504568 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.504568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.504569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.504668 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.504907 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.504916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.505110 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.505250 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.505423 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260929 03:22:42.506088 2033 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:42.506237 2033 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:42.506417 2033 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:42.506817 1573 ts_manager.cc:194] Registered new tserver with Master: f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875)
I20260929 03:22:42.507305 1573 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:42505
I20260929 03:22:42.517199 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.68.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.125.keytab.
Entry for principal oryx/127.27.68.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.125.keytab.
I20260929 03:22:42.543727 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:36901
--webserver_interface=127.27.68.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.125.keytab
--principal=oryx/127.27.68.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46757,127.27.68.125:36901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:42.625490 2038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:42.625717 2038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:42.625741 2038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:42.626845 2038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:42.626884 2038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:42.626899 2038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:42.626911 2038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:42.626924 2038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:42.628430 2038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46757,127.27.68.125:36901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.125.keytab
--principal=oryx/127.27.68.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.68.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
W20260929 03:22:42.628537 2038 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.68.126:46757,127.27.68.125:36901'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:22:42.628656 2038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:42.628839 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.633740 2044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.633740 2043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.633759 2046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.633993 2038 server_base.cc:1034] running on GCE node
I20260929 03:22:42.634251 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.634600 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.635754 2038 hybrid_clock.cc:648] HybridClock initialized: now 1790652162635728 us; error 37 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.638618 2038 init.cc:377] Logged in from keytab as oryx/127.27.68.125@KRBTEST.COM (short username oryx)
I20260929 03:22:42.639107 2038 webserver.cc:460] Webserver started at http://127.27.68.125:35693/ using document root <none> and password file <none>
I20260929 03:22:42.639371 2038 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.639421 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.639519 2038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:42.640583 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/instance:
uuid: "51ae8f015a6a474197bc50393cfb4faf"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "5a0584a5c3e57ee02d170662a0077cc0"
server_key_iv: "ad980b75d36f3c28bbf9c0a2fdf06382"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.640897 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal/instance:
uuid: "51ae8f015a6a474197bc50393cfb4faf"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "5a0584a5c3e57ee02d170662a0077cc0"
server_key_iv: "ad980b75d36f3c28bbf9c0a2fdf06382"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.642241 2038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:22:42.642961 2053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.643163 2038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:42.643321 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "51ae8f015a6a474197bc50393cfb4faf"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "5a0584a5c3e57ee02d170662a0077cc0"
server_key_iv: "ad980b75d36f3c28bbf9c0a2fdf06382"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.643404 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.654825 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.656873 2038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:42.657060 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.661165 2038 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.125:36901
I20260929 03:22:42.661204 2106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.125:36901 every 8 connection(s)
I20260929 03:22:42.661585 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
I20260929 03:22:42.662601 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:42.664263 2107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:22:42.668742 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2038
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.125@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:42.673725 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.664979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.27.68.126:46757)
0929 03:22:42.665056 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.665058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.665289 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:22:42.665742 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.665743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.665753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.665814 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.666405 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.666895 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.667976 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.668163 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.668226 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.670639 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.670640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.670640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.670649 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.672288 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.672694 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.672695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.672696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.672761 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.673025 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.673026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.673026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.673185 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.673260 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.673543 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.673629 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:22:42.673946 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.664977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41442)
0929 03:22:42.665148 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.665193 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.665643 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.665868 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.665871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.666127 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.666323 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.666326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.666981 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.666982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.667870 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.667873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.668240 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.668241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.668349 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.668921 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.668929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.670538 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.672367 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.672373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.672376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.672613 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.672830 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.672830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.672831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.672949 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.673261 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.673269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.673408 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.673512 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.673670 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
I20260929 03:22:42.674991 2107 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:22:42.678076 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.669312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.27.68.125:36901)
0929 03:22:42.669482 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.669493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.669510 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:22:42.669557 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.669559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.669576 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.669672 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.670245 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.670997 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.672142 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.672453 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.672555 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.675078 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.675085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.675085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.675093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.676866 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.677255 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.677258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.677258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.677318 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.677517 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.677518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.677519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.677660 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.677834 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.677880 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.677958 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.125@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
I20260929 03:22:42.682034 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.675374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.27.68.125:36901)
0929 03:22:42.675657 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.675659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.675667 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:42.675688 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.675689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.675699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.675778 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.676153 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.676662 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.677357 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.677570 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.677652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.679501 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.679503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.679503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.679509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.681085 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.681376 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.681377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.681377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.681421 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.681554 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.681554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.681554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.681655 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.681702 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.681872 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.681945 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260929 03:22:42.682159 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.675302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40306)
0929 03:22:42.675493 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.675502 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.675536 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.675823 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.675823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.676008 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.676064 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.676070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.676779 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.676780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.677263 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.677265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.677651 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.677652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.677720 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.678125 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.678130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.679425 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.681157 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.681158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.681158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.681320 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.681471 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.681472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.681472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.681518 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.681733 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.681734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.681775 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.681856 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.682007 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":57}
I20260929 03:22:42.683670 2107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Bootstrap starting.
I20260929 03:22:42.684367 2107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:42.684672 2107 log.cc:824] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:42.685480 2107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: No bootstrap required, opened a new log
I20260929 03:22:42.686950 2107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } }
I20260929 03:22:42.687137 2107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:42.687193 2107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ae8f015a6a474197bc50393cfb4faf, State: Initialized, Role: FOLLOWER
I20260929 03:22:42.687403 2107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } }
I20260929 03:22:42.687711 2114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } } }
I20260929 03:22:42.687754 2107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:42.687805 2114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:22:42.689587 2125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:22:42.689677 2125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:22:42.691411 2038 master_runner.cc:428] Detected that this master 51ae8f015a6a474197bc50393cfb4faf is joining an existing cluster
I20260929 03:22:42.691453 2038 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.68.125:36901
I20260929 03:22:42.700565 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.692657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.27.68.126:46757)
0929 03:22:42.692716 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.692718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.692780 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:22:42.692912 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.692913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.692930 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.692989 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.693407 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.693917 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695058 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.695399 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695446 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.697059 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.697061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.697061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.697068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.699139 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699735 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.699735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.699736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.699795 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.700042 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.700043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.700044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.700181 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.700244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.700402 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.700480 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:22:42.700564 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.692797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.27.68.125:36901)
0929 03:22:42.692835 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.692837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.692942 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:22:42.693022 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.693023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.693031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.693055 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.693441 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.694127 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695078 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.695409 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695446 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.697051 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.697052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.697053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.697060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.699268 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699684 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.699685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.699686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.699751 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699932 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.699933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.699933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.700101 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.700174 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.700330 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.700460 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260929 03:22:42.700903 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.692777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40316)
0929 03:22:42.692885 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.692895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.692952 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.693106 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.693106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.693279 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.693335 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.693336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.694215 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.694216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.694998 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.695000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.695544 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.695604 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.696274 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.696279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.696948 (+ 669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.699326 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699326 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.699327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.699598 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.699820 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.699821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.699873 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.700176 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.700177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.700232 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.700309 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.700552 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":117,"thread_start_us":62,"threads_started":1}
I20260929 03:22:42.701298 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.692542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41450)
0929 03:22:42.692700 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.692713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.692812 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.692992 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.692993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.693227 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.693309 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.693312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.694004 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.694005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.694946 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.694948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.695540 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.695541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.695604 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.696358 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.696363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.696948 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.699291 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.699292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.699619 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.699875 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.699876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.699876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.699956 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.700240 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.700241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.700280 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.700897 (+ 617us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.701226 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.126@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
I20260929 03:22:42.712239 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.704202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.27.68.125:36901)
0929 03:22:42.704263 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.704265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.704430 (+ 165us) server_negotiation.cc:407] Connection header received
0929 03:22:42.704528 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.704529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.704538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.704582 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.705155 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.705615 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.706770 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.706950 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.707005 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.709385 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.709387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.709387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.709394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.710785 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.711302 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.711303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.711303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.711355 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.711606 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.711607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.711607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.711723 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.711819 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.712037 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.712117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:42.712239 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.704152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40330)
0929 03:22:42.704313 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.704363 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.704446 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.704598 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.704602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.704989 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.705076 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.705079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.705698 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.705699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.706686 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.706689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.707028 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.707029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.707119 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.707647 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.707652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.709286 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.710871 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.710879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.710882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.711162 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.711423 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.711424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.711424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.711505 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.711813 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.711822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.711882 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.711984 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.712123 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260929 03:22:42.712852 1573 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.68.125:36901: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf"
member_type: NON_VOTER
last_known_addr {
host: "127.27.68.125"
port: 36901
}
attrs {
promote: true
}
}
dest_uuid: "0ab2e724b3f546d9b8c643c36bdc5581"
cas_config_opid_index: -1
I20260929 03:22:42.713199 1573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } }
I20260929 03:22:42.713712 2134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } }
I20260929 03:22:42.714056 2080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:42.714723 2134 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.68.125:36901
I20260929 03:22:42.714939 2038 master.cc:537] Master@127.27.68.125:36901 shutting down...
I20260929 03:22:42.714977 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } }
I20260929 03:22:42.715023 2080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:22:42.715055 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:42.715636 2135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:22:42.716727 1560 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 03:22:42.718369 1560 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 -> Peer 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125:36901): Couldn't send request to peer 51ae8f015a6a474197bc50393cfb4faf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:22:42.719115 2038 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:42.719265 2038 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:42.719326 2038 tablet_replica.cc:330] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: stopping tablet replica
I20260929 03:22:42.732493 2038 master.cc:552] Master@127.27.68.125:36901 shutdown complete.
I20260929 03:22:42.732549 2038 master_runner.cc:305] Clearing existing system tablet
I20260929 03:22:42.732702 2038 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.732754 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.733415 2038 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260929 03:22:42.733947 2139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.734074 2038 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.734149 2038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:42.734180 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "51ae8f015a6a474197bc50393cfb4faf"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "5a0584a5c3e57ee02d170662a0077cc0"
server_key_iv: "ad980b75d36f3c28bbf9c0a2fdf06382"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.734845 2038 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:42.736372 2038 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:22:42.736461 2038 log.cc:1196] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:22:42.736678 2038 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Deleting consensus metadata
I20260929 03:22:42.736802 2038 master_runner.cc:315] Copying system tablet from 127.27.68.126:46757
I20260929 03:22:42.737422 2038 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: tablet copy: Beginning tablet copy session from remote peer at address 127.27.68.126:46757
I20260929 03:22:42.744228 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.737775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41456)
0929 03:22:42.737917 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.737928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.738035 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.738247 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.738248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.738412 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.738468 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.738469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.739454 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.739454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.740008 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.740009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.740476 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.740476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.740525 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.741045 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.741051 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.741377 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.743133 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.743133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.743134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.743373 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.743600 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.743600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.743601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.743655 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.743934 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.743935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.743973 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.744051 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.744140 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260929 03:22:42.744266 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.737836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.27.68.126:46757)
0929 03:22:42.737909 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.737912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.738021 (+ 109us) server_negotiation.cc:407] Connection header received
0929 03:22:42.738153 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.738154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.738167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.738243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.738541 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.739367 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.740093 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.740396 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.740472 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.741471 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.741473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.741473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.741480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.743045 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.743445 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.743446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.743447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.743530 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.743716 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.743717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.743718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.743869 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.743944 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.744080 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.744168 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:42.744614 1594 tablet_copy_service.cc:140] P 0ab2e724b3f546d9b8c643c36bdc5581: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 51ae8f015a6a474197bc50393cfb4faf ({username='oryx', principal='oryx/127.27.68.125@KRBTEST.COM'} at 127.0.0.1:41456)
I20260929 03:22:42.744728 1594 tablet_copy_service.cc:161] P 0ab2e724b3f546d9b8c643c36bdc5581: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 51ae8f015a6a474197bc50393cfb4faf at {username='oryx', principal='oryx/127.27.68.125@KRBTEST.COM'} at 127.0.0.1:41456: session id = 51ae8f015a6a474197bc50393cfb4faf-00000000000000000000000000000000
I20260929 03:22:42.745401 1594 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:22:42.746090 2038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:22:42.747521 2038 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: tablet copy: Starting download of 0 data blocks...
I20260929 03:22:42.747689 2038 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: tablet copy: Starting download of 1 WAL segments...
I20260929 03:22:42.748709 2038 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:22:42.749775 1594 tablet_copy_service.cc:342] P 0ab2e724b3f546d9b8c643c36bdc5581: Request end of tablet copy session 51ae8f015a6a474197bc50393cfb4faf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.68.125@KRBTEST.COM'} at 127.0.0.1:41456
I20260929 03:22:42.749846 1594 tablet_copy_service.cc:433] P 0ab2e724b3f546d9b8c643c36bdc5581: ending tablet copy session 51ae8f015a6a474197bc50393cfb4faf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 51ae8f015a6a474197bc50393cfb4faf
I20260929 03:22:42.752578 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.753832 2146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.753899 2149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.753818 2147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.756273 2038 server_base.cc:1034] running on GCE node
I20260929 03:22:42.756387 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.756543 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.757684 2038 hybrid_clock.cc:648] HybridClock initialized: now 1790652162757677 us; error 27 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.759070 2038 init.cc:377] Logged in from keytab as oryx/127.27.68.125@KRBTEST.COM (short username oryx)
I20260929 03:22:42.759402 2038 webserver.cc:460] Webserver started at http://127.27.68.125:36953/ using document root <none> and password file <none>
I20260929 03:22:42.759536 2038 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.759583 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.760236 2038 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:42.760795 2156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.760955 2038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:42.761016 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "51ae8f015a6a474197bc50393cfb4faf"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "5a0584a5c3e57ee02d170662a0077cc0"
server_key_iv: "ad980b75d36f3c28bbf9c0a2fdf06382"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.761080 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.766086 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.767951 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.771169 2038 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.125:36901
I20260929 03:22:42.771664 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
I20260929 03:22:42.771670 2210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.125:36901 every 8 connection(s)
I20260929 03:22:42.772617 2211 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:22:42.773105 2211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Bootstrap starting.
I20260929 03:22:42.776098 2211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:22:42.776351 2211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: Bootstrap complete.
I20260929 03:22:42.776580 2211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } }
I20260929 03:22:42.776675 2211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51ae8f015a6a474197bc50393cfb4faf, State: Initialized, Role: LEARNER
I20260929 03:22:42.776715 2211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } }
I20260929 03:22:42.776901 2211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:42.776896 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } }
I20260929 03:22:42.777024 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: LEARNER
I20260929 03:22:42.778554 2225 catalog_manager.cc:1259] Loaded cluster ID: f56b3d4201c04b10a17fc48e60d4415a
I20260929 03:22:42.778637 2225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: loading cluster ID for follower catalog manager: success
I20260929 03:22:42.779712 2225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: acquiring CA information for follower catalog manager: success
I20260929 03:22:42.780176 2225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:22:42.786757 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.779824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.27.68.125:36901)
0929 03:22:42.779935 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.779937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.779948 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:42.779976 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.779977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.779988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.780070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.780786 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.781392 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.782434 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.782627 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.782674 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.784141 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.784143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.784143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.784150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.785618 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.786024 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.786024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.786025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.786074 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.786284 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.786285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.786285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.786384 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.786455 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.786620 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.786686 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.27.68.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.124.keytab.
Entry for principal oryx/127.27.68.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.124.keytab.
I20260929 03:22:42.815936 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:38451
--webserver_interface=127.27.68.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.124.keytab
--principal=oryx/127.27.68.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:42.899999 2229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:42.900223 2229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:42.900274 2229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:42.901455 2229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:42.901499 2229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:42.901513 2229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:42.901526 2229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:42.901538 2229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:42.902980 2229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.124.keytab
--principal=oryx/127.27.68.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.68.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:42.903196 2229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:42.903463 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:42.908288 2237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:42.908298 2234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.908594 2229 server_base.cc:1034] running on GCE node
W20260929 03:22:42.908360 2235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:42.908888 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:42.909214 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:42.910385 2229 hybrid_clock.cc:648] HybridClock initialized: now 1790652162910379 us; error 37 us; skew 500 ppm
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:42.913043 2229 init.cc:377] Logged in from keytab as oryx/127.27.68.124@KRBTEST.COM (short username oryx)
I20260929 03:22:42.913525 2229 webserver.cc:460] Webserver started at http://127.27.68.124:40591/ using document root <none> and password file <none>
I20260929 03:22:42.913753 2229 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:42.913820 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:42.913969 2229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:42.915050 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/instance:
uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "6aa39291b8b3326037ef1573e32cda58"
server_key_iv: "2e598bbb920e5bfdffec742f5c9613d6"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.915524 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal/instance:
uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "6aa39291b8b3326037ef1573e32cda58"
server_key_iv: "2e598bbb920e5bfdffec742f5c9613d6"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.916875 2229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:42.917609 2244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:42.917778 2229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:42.917881 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "6aa39291b8b3326037ef1573e32cda58"
server_key_iv: "2e598bbb920e5bfdffec742f5c9613d6"
server_key_version: "encryptionkey@0"
I20260929 03:22:42.917965 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:42.933070 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:42.935276 2229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:42.935453 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:42.939663 2229 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.124:38451
I20260929 03:22:42.939697 2296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.124:38451 every 8 connection(s)
I20260929 03:22:42.940058 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
I20260929 03:22:42.940908 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2229
I20260929 03:22:42.941140 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:42.943517 2297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.124@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:42.951508 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.944288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.27.68.126:46757)
0929 03:22:42.944355 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.944357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.944498 (+ 141us) server_negotiation.cc:407] Connection header received
0929 03:22:42.944569 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.944570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.944582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.944645 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.945075 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.945616 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.946422 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.946765 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.946847 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.948899 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.948900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.948901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.948909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.950141 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950542 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.950543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.950543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.950608 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950858 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.950859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.950860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.950992 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.951071 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.951321 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.951406 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:22:42.951738 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.941665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.27.68.124:38451)
0929 03:22:42.941842 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.941851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.941887 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:22:42.942353 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.942362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.942430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.942596 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.943173 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.944429 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.945262 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.945679 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.945787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.948491 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.948500 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.948500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.948507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.950196 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950590 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.950593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.950594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.950694 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950906 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.950907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.950908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.951126 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.951329 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.951374 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.951569 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260929 03:22:42.951745 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.944179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41462)
0929 03:22:42.944373 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.944414 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.944497 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.944677 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.944680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.944932 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.944993 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.944994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.945710 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.945711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.946335 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.946337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.946871 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.946872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.946919 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.947418 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.947427 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.948805 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.950247 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.950256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.950462 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.950685 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.950686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.950686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.950784 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.951061 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.951067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.951147 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.951295 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.951470 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
I20260929 03:22:42.952556 2297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.124@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
I20260929 03:22:42.958784 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.952820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40360)
0929 03:22:42.952872 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.952882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.952919 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.953065 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.953065 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.953180 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.953230 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.953232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.953939 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.953939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.954434 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.954435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.954769 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.954770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.954821 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.955196 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.955200 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.956437 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.957835 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.957836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.957836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.958012 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.958193 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.958193 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.958194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.958245 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.958480 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.958481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.958548 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.958620 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.958712 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:22:42.958784 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.952892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.27.68.125:36901)
0929 03:22:42.952932 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.952934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.952971 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:22:42.952995 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.952996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.953008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.953062 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.953295 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.953863 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.954508 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.954699 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.954757 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.956529 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.956531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.956531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.956539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.957748 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.958081 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.958081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.958082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.958135 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.958298 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.958298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.958299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.958397 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.958447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.958639 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.958717 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:42.959334 2297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:22:42 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652162, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.124@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
I20260929 03:22:42.965842 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.959697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.27.68.124:38451)
0929 03:22:42.959732 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.959734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.959743 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:42.959764 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.959764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.959774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.959828 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.960081 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.960602 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.961310 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.961505 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.961585 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.963442 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.963444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.963444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.963451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.964641 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.965000 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.965001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.965002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.965062 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.965253 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.965254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.965254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.965372 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.965443 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.965611 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.965763 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":20}
I20260929 03:22:42.965840 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.959620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56498)
0929 03:22:42.959671 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.959683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.959721 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.959830 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.959831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.959973 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.960004 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.960005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.960696 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.960698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.961214 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.961215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.961581 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.961581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.961628 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.962092 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.962096 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.963352 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.964739 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.964740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.964741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.964909 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.965127 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.965127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.965128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.965180 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.965445 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.965446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.965490 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.965590 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.965716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":3}
I20260929 03:22:42.967800 2297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Bootstrap starting.
I20260929 03:22:42.968478 2297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:42.968770 2297 log.cc:824] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:42.969547 2297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: No bootstrap required, opened a new log
I20260929 03:22:42.971113 2297 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } }
I20260929 03:22:42.971380 2297 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:42.971439 2297 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfeb4ce56f364dff8ab2f996b722b71c, State: Initialized, Role: FOLLOWER
I20260929 03:22:42.971585 2297 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } }
I20260929 03:22:42.971920 2302 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } } }
I20260929 03:22:42.972000 2297 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:42.972029 2302 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:22:42.973654 2313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:22:42.973719 2313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:22:42.976074 2229 master_runner.cc:428] Detected that this master cfeb4ce56f364dff8ab2f996b722b71c is joining an existing cluster
I20260929 03:22:42.976137 2229 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.68.124:38451
I20260929 03:22:42.984536 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.27.68.124:38451)
0929 03:22:42.977442 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.977444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.977766 (+ 322us) server_negotiation.cc:407] Connection header received
0929 03:22:42.977796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.977797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.977810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.977872 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.978364 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979108 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980179 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980473 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980516 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.981795 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:42.981796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:42.981796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:42.981802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:42.983393 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.983826 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.983827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.983827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.983877 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:42.984073 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:42.984073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:42.984073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:42.984173 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:42.984222 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:42.984404 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:22:42.984467 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
I20260929 03:22:42.984668 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56500)
0929 03:22:42.977690 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.977699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.977753 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.977912 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.977912 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.978194 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.978274 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.978277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979203 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.979204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.980083 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.980086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980592 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.980650 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:42.981183 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:42.981189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:42.981694 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:42.983479 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.983480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.983481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.983743 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.983939 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:42.983940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:42.983940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:42.984011 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:42.984243 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:42.984244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:42.984295 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:22:42.984390 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:22:42.984536 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":39,"spinlock_wait_cycles":9984,"thread_start_us":171,"threads_started":1}
I20260929 03:22:43.003119 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1636
I20260929 03:22:43.004302 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.27.68.125:36901)
0929 03:22:42.977643 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.977645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.977892 (+ 247us) server_negotiation.cc:407] Connection header received
0929 03:22:42.978039 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.978040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.978052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.978083 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.978522 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979325 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980548 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980859 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980901 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.000825 (+ 19924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.000828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.000828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.000839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.002810 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.003383 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.003385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.003385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.003446 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.003644 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.003645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.003645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.003790 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.003850 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.004090 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.004192 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:22:43.004640 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40372)
0929 03:22:42.977842 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.977851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.977941 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.978148 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.978149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.978373 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.978425 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.978426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979427 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.979428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.980436 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.980439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980980 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.981046 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.000101 (+ 19055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.000107 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.000663 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.002936 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.002937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.002938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.003287 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.003515 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.003516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.003516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.003573 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.003903 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.003904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.003956 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.004388 (+ 432us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.004516 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":57,"thread_start_us":69,"threads_started":1}
I20260929 03:22:43.006186 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.27.68.126:46757)
0929 03:22:42.977486 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:42.977488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:42.977680 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:22:42.977811 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:42.977813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:42.977827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:42.977864 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:42.978505 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979311 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980548 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980859 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980901 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.001116 (+ 20215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.001120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.001121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.001153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.002824 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.005198 (+ 2374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.005200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.005200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.005276 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.005525 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.005526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.005526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.005673 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.005732 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.005934 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.006028 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:22:43.006608 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:42.977295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41468)
0929 03:22:42.977544 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:42.977556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:42.977727 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:42.977978 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:42.977979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:42.978335 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:42.978404 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.978406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.979427 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.979428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.980436 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:42.980439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:42.980993 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:42.980993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:42.981046 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.000690 (+ 19644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.000695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.000998 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.002918 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.002919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.002919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.005061 (+ 2142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.005388 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.005389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.005390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.005456 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.005778 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.005779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.005826 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.006310 (+ 484us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.006443 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":56,"thread_start_us":88,"threads_started":1}
I20260929 03:22:43.008845 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:37789
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:46757
--webserver_port=37701
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652161, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.126@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
I20260929 03:22:43.019387 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.009248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56506)
0929 03:22:43.009300 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.009314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.009370 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.010592 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.010593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.010791 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.010849 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.010851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.011931 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.011933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.012913 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.012917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.013554 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.013555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.013653 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.014169 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.014175 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.015946 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.018051 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.018053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.018053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.018302 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.018605 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.018606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.018606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.018674 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.018973 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.018974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.019023 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.019106 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.019268 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:22:43.020011 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.010331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.27.68.124:38451)
0929 03:22:43.010397 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.010399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.010408 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:43.010447 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.010448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.010459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.010520 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.010943 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.011833 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.013016 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.013397 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.013493 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.016056 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.016057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.016058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.016081 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.017940 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.018405 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.018406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.018407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.018522 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.018746 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.018747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.018747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.018886 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.018947 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.019821 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.019908 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:22:43.020749 1573 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.68.124:38451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
member_type: NON_VOTER
last_known_addr {
host: "127.27.68.124"
port: 38451
}
attrs {
promote: true
}
}
dest_uuid: "0ab2e724b3f546d9b8c643c36bdc5581"
cas_config_opid_index: 5
I20260929 03:22:43.021010 1573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } }
I20260929 03:22:43.021428 2135 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.022173 2271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:43.022507 2135 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.68.124:38451
I20260929 03:22:43.022588 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.022677 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.023043 2229 master.cc:537] Master@127.27.68.124:38451 shutting down...
I20260929 03:22:43.023145 2271 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:22:43.023627 2134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:22:43.025326 1561 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 -> Peer cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124:38451): Couldn't send request to peer cfeb4ce56f364dff8ab2f996b722b71c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:22:43.027500 2229 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:43.027611 2229 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:43.027659 2229 tablet_replica.cc:330] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: stopping tablet replica
I20260929 03:22:43.028638 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.021513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40380)
0929 03:22:43.023592 (+ 2079us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.023604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.023642 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.024042 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.024043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.024044 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.024101 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.024103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.025169 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.025170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.026873 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.026879 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.028200 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.028201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.028276 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.028458 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.028508 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.028556 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2060}
I20260929 03:22:43.028955 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.022845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.27.68.125:36901)
0929 03:22:43.023815 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.023816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.023824 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:43.023847 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.023848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.023870 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.023927 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.024172 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.024933 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.026996 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.027777 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.028046 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.028137 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.028847 (+ 710us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.028892 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946}
I20260929 03:22:43.029729 2184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 LEARNER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:22:43.029963 2134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:22:43.030612 2184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.032023 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab2e724b3f546d9b8c643c36bdc5581. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.032125 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: LEARNER
I20260929 03:22:43.032160 2214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: NON_VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: true } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.032202 2214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: LEARNER
I20260929 03:22:43.032454 2134 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: attempting to promote NON_VOTER 51ae8f015a6a474197bc50393cfb4faf to VOTER
I20260929 03:22:43.032671 2134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } }
I20260929 03:22:43.033277 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.033138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56522)
0929 03:22:43.033190 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.033240 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:22:43.033655 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.033555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56524)
0929 03:22:43.033608 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.033636 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:22:43.033761 2184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 LEARNER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 03:22:43.033987 1561 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 -> Peer cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124:38451): Couldn't send request to peer cfeb4ce56f364dff8ab2f996b722b71c. Status: Network error: Client connection negotiation failed: client connection to 127.27.68.124:38451: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:22:43.034157 2135 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 03:22:43.034866 2325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.034966 2184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.035984 2324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.036082 2324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:22:43.036315 2135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.036402 2135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.036504 2325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.036561 2325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.040818 2229 master.cc:552] Master@127.27.68.124:38451 shutdown complete.
I20260929 03:22:43.040879 2229 master_runner.cc:305] Clearing existing system tablet
I20260929 03:22:43.041024 2229 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:43.041080 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:43.041903 2229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.042492 2328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.042687 2229 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:43.042752 2229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:22:43.042780 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "6aa39291b8b3326037ef1573e32cda58"
server_key_iv: "2e598bbb920e5bfdffec742f5c9613d6"
server_key_version: "encryptionkey@0"
I20260929 03:22:43.043802 2229 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:43.045269 2229 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:22:43.045387 2229 log.cc:1196] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:22:43.045607 2229 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Deleting consensus metadata
I20260929 03:22:43.045744 2229 master_runner.cc:315] Copying system tablet from 127.27.68.126:46757
I20260929 03:22:43.046423 2229 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.68.126:46757
I20260929 03:22:43.054390 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.046742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:41478)
0929 03:22:43.046937 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.046950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.047033 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.047200 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.047201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.047465 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.047534 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.047536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.048520 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.048522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.049519 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.049522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.050017 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.050018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.050102 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.050640 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.050646 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.051098 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.053148 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.053149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.053149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.053407 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.053635 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.053636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.053637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.053711 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.053991 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.053992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.054046 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.054134 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.054258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260929 03:22:43.054419 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.046826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.27.68.126:46757)
0929 03:22:43.046880 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.046883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.047040 (+ 157us) server_negotiation.cc:407] Connection header received
0929 03:22:43.047108 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.047109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.047123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.047178 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.047629 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.048419 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.049624 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.049928 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.050002 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.051201 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.051202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.051203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.051211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.053045 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.053492 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.053494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.053494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.053559 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.053783 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.053784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.053784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.053921 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.053988 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.054173 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.054259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:22:43.054852 1594 tablet_copy_service.cc:140] P 0ab2e724b3f546d9b8c643c36bdc5581: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cfeb4ce56f364dff8ab2f996b722b71c ({username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:41478)
I20260929 03:22:43.054903 1594 tablet_copy_service.cc:161] P 0ab2e724b3f546d9b8c643c36bdc5581: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cfeb4ce56f364dff8ab2f996b722b71c at {username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:41478: session id = cfeb4ce56f364dff8ab2f996b722b71c-00000000000000000000000000000000
I20260929 03:22:43.055562 1594 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:22:43.056077 2229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:22:43.057497 2229 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: tablet copy: Starting download of 0 data blocks...
I20260929 03:22:43.057649 2229 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: tablet copy: Starting download of 1 WAL segments...
I20260929 03:22:43.058532 2229 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:22:43.059607 1594 tablet_copy_service.cc:342] P 0ab2e724b3f546d9b8c643c36bdc5581: Request end of tablet copy session cfeb4ce56f364dff8ab2f996b722b71c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:41478
I20260929 03:22:43.059670 1594 tablet_copy_service.cc:433] P 0ab2e724b3f546d9b8c643c36bdc5581: ending tablet copy session cfeb4ce56f364dff8ab2f996b722b71c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cfeb4ce56f364dff8ab2f996b722b71c
I20260929 03:22:43.062690 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:43.063949 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.064160 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.068179 2229 server_base.cc:1034] running on GCE node
W20260929 03:22:43.068382 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.068589 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:43.068750 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:43.069871 2229 hybrid_clock.cc:648] HybridClock initialized: now 1790652163069863 us; error 30 us; skew 500 ppm
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:43.071563 2229 init.cc:377] Logged in from keytab as oryx/127.27.68.124@KRBTEST.COM (short username oryx)
I20260929 03:22:43.071834 2229 webserver.cc:460] Webserver started at http://127.27.68.124:33393/ using document root <none> and password file <none>
I20260929 03:22:43.071950 2229 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:43.071986 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:43.072762 2229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.073395 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.073582 2229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.073634 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "6aa39291b8b3326037ef1573e32cda58"
server_key_iv: "2e598bbb920e5bfdffec742f5c9613d6"
server_key_version: "encryptionkey@0"
I20260929 03:22:43.073697 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:43.097256 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:43.099383 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:43.102397 2229 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.124:38451
I20260929 03:22:43.102510 2397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.124:38451 every 8 connection(s)
I20260929 03:22:43.102756 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
I20260929 03:22:43.103960 2398 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:22:43.104468 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Bootstrap starting.
I20260929 03:22:43.107630 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:22:43.107853 2398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Bootstrap complete.
I20260929 03:22:43.108034 2398 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } }
I20260929 03:22:43.108148 2398 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cfeb4ce56f364dff8ab2f996b722b71c, State: Initialized, Role: LEARNER
I20260929 03:22:43.108188 2398 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } }
I20260929 03:22:43.108369 2398 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:43.108374 2401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.108647 2401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: LEARNER
I20260929 03:22:43.109827 2412 catalog_manager.cc:1259] Loaded cluster ID: f56b3d4201c04b10a17fc48e60d4415a
I20260929 03:22:43.110713 2412 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: loading cluster ID for follower catalog manager: success
I20260929 03:22:43.112048 2412 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: acquiring CA information for follower catalog manager: success
I20260929 03:22:43.112434 2412 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:22:43.119446 2322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:43.119685 2322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:43.119727 2322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:43.120970 2322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:43.121040 2322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:43.121089 2322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:43.122812 2322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:37789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=37701
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:43.123104 2322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:43.123363 2322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:43.129746 2417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.129727 2418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.129832 2420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.129881 2322 server_base.cc:1034] running on GCE node
I20260929 03:22:43.130192 2322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:43.130496 2322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:43.131661 2322 hybrid_clock.cc:648] HybridClock initialized: now 1790652163131647 us; error 34 us; skew 500 ppm
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:43.134492 2322 init.cc:377] Logged in from keytab as oryx/127.27.68.65@KRBTEST.COM (short username oryx)
I20260929 03:22:43.135071 2322 webserver.cc:460] Webserver started at http://127.27.68.65:37701/ using document root <none> and password file <none>
I20260929 03:22:43.135380 2322 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:43.135468 2322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:43.136974 2322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.137615 2427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.137825 2322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.137918 2322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "5ece50a719974528abcafc4a287fe9c2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "644db21c83a74aa5f206fe7521f62471"
server_key_iv: "28825d4e9aeb3f95eb8dcbd21044ec27"
server_key_version: "encryptionkey@0"
I20260929 03:22:43.138278 2322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:43.159425 2322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:43.161513 2322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:43.161715 2322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:43.161957 2322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:43.162286 2322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:43.162320 2322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.162343 2322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:43.162376 2322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.168347 2322 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:37789
I20260929 03:22:43.168395 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:37789 every 8 connection(s)
I20260929 03:22:43.168764 2322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
I20260929 03:22:43.176359 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2322
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:43.176525 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1770
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
I20260929 03:22:43.180838 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34259 (local address 127.27.68.125:36901)
0929 03:22:43.169556 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.169559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.169779 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:22:43.170400 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.170402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.170414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.170468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.171458 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.171958 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173370 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173665 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173723 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.177171 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.177172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.177173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.177182 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.179093 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.179651 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.179653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.179653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.179717 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.179992 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.179993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.179993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.180167 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.180219 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.180594 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.180689 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:22:43.181507 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.27.68.65:34259)
0929 03:22:43.169713 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.169724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.170294 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.170879 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.170879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.171082 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.171365 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.171370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.172076 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.172077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173242 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.173245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173771 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173864 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.174635 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.174640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.176484 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.179188 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.179196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.179200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.179558 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.179786 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.179787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.179787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.179926 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.180251 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.180258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.180439 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.180985 (+ 546us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.181207 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":9,"thread_start_us":91,"threads_started":1}
I20260929 03:22:43.181721 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:37325 (local address 127.27.68.126:46757)
0929 03:22:43.169945 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.169947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.169957 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:43.170527 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.170528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.170540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.170600 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.171483 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.172286 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173402 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173635 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.178336 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.178338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.178338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.178348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.180513 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180891 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.180891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.180892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.180944 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.181169 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.181170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.181170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.181299 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.181340 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.181532 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.181616 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:43.181965 2543 heartbeater.cc:347] Connected to a master server at 127.27.68.125:36901
I20260929 03:22:43.182080 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:37325)
0929 03:22:43.169898 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.169907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.170293 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.170632 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.170633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.171084 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.171365 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.171371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.172368 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.172369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173244 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.173246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173806 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173867 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.174586 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.174595 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.178233 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.180593 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.180594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.180809 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.181018 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.181018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.181019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.181094 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.181385 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.181386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.181422 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.181795 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.181920 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":35,"thread_start_us":55,"threads_started":1}
I20260929 03:22:43.182104 2543 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.182120 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40475 (local address 127.27.68.124:38451)
0929 03:22:43.169776 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.169778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.169787 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:43.170413 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.170414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.170428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.170479 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.171481 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.172478 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173296 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173606 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173676 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.178380 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.178382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.178382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.178390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.180147 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180546 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.180548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.180548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.180625 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180825 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.180825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.180826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.180964 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.181007 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.181484 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.182020 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":48,"threads_started":1}
I20260929 03:22:43.182304 2543 heartbeater.cc:511] Master 127.27.68.125:36901 requested a full tablet report, sending...
I20260929 03:22:43.182643 2542 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:43.182693 2542 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.182763 2542 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:43.182797 2174 ts_manager.cc:194] Registered new tserver with Master: 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:43.183046 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.169488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.27.68.65:40475)
0929 03:22:43.169643 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.169689 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.170292 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.170547 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.170550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.171080 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.171365 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.171371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.172552 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.172553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173199 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.173201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.173699 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.173700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.173867 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.174739 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.174744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.178034 (+ 3290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.180245 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.180246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.180462 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.180699 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.180700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.180700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.180753 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.181306 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.181306 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.181350 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.182760 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.182951 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":14,"thread_start_us":86,"threads_started":1}
I20260929 03:22:43.183036 1573 ts_manager.cc:194] Re-registered known tserver with Master: 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:43.183390 2541 heartbeater.cc:347] Connected to a master server at 127.27.68.124:38451
I20260929 03:22:43.183440 2541 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.183502 2541 heartbeater.cc:511] Master 127.27.68.124:38451 requested a full tablet report, sending...
I20260929 03:22:43.183568 1573 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:37325
I20260929 03:22:43.183841 2362 ts_manager.cc:194] Registered new tserver with Master: 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:43.184327 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:42355
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:46757
--webserver_port=33083
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:43.267153 2549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:43.267480 2549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:43.267524 2549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:43.268874 2549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:43.268944 2549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:43.268991 2549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:43.270629 2549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:42355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=33083
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:43.270944 2549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:43.271265 2549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:43.276600 2557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.276646 2554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.276607 2555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.276819 2549 server_base.cc:1034] running on GCE node
I20260929 03:22:43.277045 2549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:43.277398 2549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:43.278590 2549 hybrid_clock.cc:648] HybridClock initialized: now 1790652163278568 us; error 51 us; skew 500 ppm
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:43.281281 2549 init.cc:377] Logged in from keytab as oryx/127.27.68.66@KRBTEST.COM (short username oryx)
I20260929 03:22:43.281728 2549 webserver.cc:460] Webserver started at http://127.27.68.66:33083/ using document root <none> and password file <none>
I20260929 03:22:43.281934 2549 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:43.281991 2549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:43.283522 2549 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:43.284116 2564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.284319 2549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.284412 2549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "73d33d0f582f4413958e5fb2dcc6428e"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "c980b7aaac6d0e08d119718c6c935b80"
server_key_iv: "bd9b53fa77dad478b09521217aa8d1be"
server_key_version: "encryptionkey@0"
I20260929 03:22:43.284766 2549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:43.315208 2549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:43.317209 2549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:43.317327 2549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:43.317519 2549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:43.317847 2549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:43.317879 2549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.317901 2549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:43.317945 2549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.323885 2549 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42355
I20260929 03:22:43.323971 2677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42355 every 8 connection(s)
I20260929 03:22:43.324342 2549 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:43.329460 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2549
I20260929 03:22:43.329649 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1904
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
I20260929 03:22:43.336941 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:38875
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:46757
--webserver_port=46199
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:32869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:22:43.337514 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:54659 (local address 127.27.68.125:36901)
0929 03:22:43.325588 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.325590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.325681 (+ 91us) server_negotiation.cc:407] Connection header received
0929 03:22:43.326642 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.326643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.326657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.326752 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.328205 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328658 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.329689 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.330025 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.330068 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.333542 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.333544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.333544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.333554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.335671 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336240 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.336241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.336242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.336303 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336619 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.336621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.336621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.336774 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.336856 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.337302 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.337392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:22:43.337571 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:59463 (local address 127.27.68.124:38451)
0929 03:22:43.325263 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.325264 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.325483 (+ 219us) server_negotiation.cc:407] Connection header received
0929 03:22:43.326659 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.326660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.326677 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.326709 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.327610 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328119 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.329678 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.329983 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.330022 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.334257 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.334258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.334259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.334266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.336012 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336414 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.336416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.336416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.336471 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336682 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.336683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.336684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.336838 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.336920 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.337318 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.337428 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:22:43.338140 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.27.68.66:54659)
0929 03:22:43.325627 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.325636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.326538 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.326850 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.326851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.327272 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.327495 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.327501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328769 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.328770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.329575 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.329579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.330145 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.330145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.330266 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.331101 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.331106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.333428 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.335768 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.335777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.335780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.336134 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.336375 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.336376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.336524 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.336857 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.336866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.337110 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.337626 (+ 516us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.337851 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":22,"thread_start_us":66,"threads_started":1}
I20260929 03:22:43.338177 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.27.68.66:59463)
0929 03:22:43.325347 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.325385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.326525 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.326838 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.326843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.327244 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.327496 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.327502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328233 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.328234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.329569 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.329572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.331282 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.331283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.331359 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.331786 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.331791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.334160 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.336120 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.336121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.336328 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.336544 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.336545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.336623 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.336965 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.336966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.337096 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.337716 (+ 620us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.337944 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":130,"thread_start_us":113,"threads_started":1}
I20260929 03:22:43.338670 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:58997 (local address 127.27.68.126:46757)
0929 03:22:43.325205 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.325207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.325654 (+ 447us) server_negotiation.cc:407] Connection header received
0929 03:22:43.326660 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.326661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.326675 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.326709 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.327620 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328121 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.329678 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.329979 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.330022 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.334047 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.334049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.334050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.334059 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.336460 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.337384 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.337385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.337386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.337448 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.338042 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.338043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.338043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.338199 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.338247 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.338452 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.338544 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:43.339020 2680 heartbeater.cc:347] Connected to a master server at 127.27.68.125:36901
I20260929 03:22:43.339098 2678 heartbeater.cc:347] Connected to a master server at 127.27.68.124:38451
I20260929 03:22:43.339114 2680 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.339131 2678 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.339334 2678 heartbeater.cc:511] Master 127.27.68.124:38451 requested a full tablet report, sending...
I20260929 03:22:43.339334 2680 heartbeater.cc:511] Master 127.27.68.125:36901 requested a full tablet report, sending...
I20260929 03:22:43.339763 2174 ts_manager.cc:194] Registered new tserver with Master: 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355)
I20260929 03:22:43.339780 2362 ts_manager.cc:194] Registered new tserver with Master: 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355)
I20260929 03:22:43.340051 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.325155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:58997)
0929 03:22:43.325591 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.325601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.326526 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.326838 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.326843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.327267 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.327496 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.327502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.328233 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.328234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.329570 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.329572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.330118 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.330119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.330265 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.330988 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.330999 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.333909 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.336557 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.336558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.336558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.337298 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.337517 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.337518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.337518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.337954 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.338301 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.338301 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.338341 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.338408 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.339450 (+ 1042us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":62,"thread_start_us":108,"threads_started":1}
I20260929 03:22:43.340354 2679 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:43.340404 2679 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.340490 2679 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:43.340750 1573 ts_manager.cc:194] Re-registered known tserver with Master: 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355)
I20260929 03:22:43.341100 1573 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:58997
W20260929 03:22:43.422446 2685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:43.422721 2685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:43.422761 2685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:43.423976 2685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:43.424053 2685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:43.424098 2685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:43.425644 2685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:32869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:38875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=46199
--tserver_master_addrs=127.27.68.126:46757,127.27.68.125:36901,127.27.68.124:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:43.425932 2685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:43.426184 2685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:43.431257 2690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.431286 2685 server_base.cc:1034] running on GCE node
W20260929 03:22:43.431257 2693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:43.431253 2691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:43.431604 2685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:43.431911 2685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:43.433081 2685 hybrid_clock.cc:648] HybridClock initialized: now 1790652163433073 us; error 36 us; skew 500 ppm
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:43.435825 2685 init.cc:377] Logged in from keytab as oryx/127.27.68.67@KRBTEST.COM (short username oryx)
I20260929 03:22:43.436492 2685 webserver.cc:460] Webserver started at http://127.27.68.67:46199/ using document root <none> and password file <none>
I20260929 03:22:43.436760 2685 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:43.436866 2685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:43.438999 2685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:43.439630 2700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.439792 2685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.439893 2685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "f82b909bbac847c5bbca02862f1e3bf2"
format_stamp: "Formatted at 2026-09-29 03:22:42 on dist-test-slave-fzwk"
server_key: "68dd5d5a130b34bb57cf43e403886e95"
server_key_iv: "e2cc2b3135946ca0fa3ec027218a1c67"
server_key_version: "encryptionkey@0"
I20260929 03:22:43.440271 2685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:43.467736 2685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:43.469703 2685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:43.469887 2685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:43.470146 2685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:43.470489 2685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:43.470544 2685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.470587 2685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:43.470618 2685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:43.477092 2685 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:38875
I20260929 03:22:43.477133 2814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:38875 every 8 connection(s)
I20260929 03:22:43.477627 2685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:43.483753 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2685
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.124@KRBTEST.COM
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.125@KRBTEST.COM
Sep 29 03:22:43 dist-test-slave-fzwk krb5kdc[1528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652163, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:43.489488 2822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.478952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:33743)
0929 03:22:43.479053 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.479065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.479377 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.479641 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.479646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.480092 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.480266 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.480272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.481238 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.481239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.482181 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.482183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.483294 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.483295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.483363 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.483792 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.483798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.485803 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.487807 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.487808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.487808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.488177 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.488382 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.488382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.488383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.488513 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.488783 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.488789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.488967 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.489076 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.489249 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":48,"thread_start_us":63,"threads_started":1}
I20260929 03:22:43.489838 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.478393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.27.68.67:59627)
0929 03:22:43.478580 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.478644 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.479360 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.479671 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.479672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.480031 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.480264 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.480269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.480997 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.480998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.481832 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.481833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.482276 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.482444 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.482999 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.483007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.485862 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.488662 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.488663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.488664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.488978 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.489173 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.489173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.489174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.489221 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.489486 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.489486 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.489532 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.489603 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.489738 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":13,"thread_start_us":89,"threads_started":1}
I20260929 03:22:43.489863 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.478830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:40455 (local address 127.27.68.125:36901)
0929 03:22:43.478879 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.478881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.478952 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:22:43.479463 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.479464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.479477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.479532 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.480379 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.481137 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482290 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.482607 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482650 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.485486 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.485488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.485488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.485496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.487672 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.488997 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.488998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.488998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.489059 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.489269 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.489270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.489270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.489399 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.489459 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.489703 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.489775 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:22:43.489938 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.479136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:33743 (local address 127.27.68.126:46757)
0929 03:22:43.479206 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.479208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.479262 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:22:43.479475 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.479476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.479487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.479519 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.480379 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.481142 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482290 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.483149 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.483198 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.485899 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.485901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.485901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.485926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.487713 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.488254 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.488256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.488256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.488316 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.488577 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.488578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.488578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.488703 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.488757 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.489758 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.489845 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:43.490181 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.478769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.27.68.67:40455)
0929 03:22:43.478898 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.478908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.479378 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.479989 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.479989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.480165 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:43.480266 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.480272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.481238 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.481239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.482168 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.482170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.482727 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.482786 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.483167 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:43.483172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:43.485388 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:43.487782 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.487791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.487793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.488869 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.489130 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:43.489131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:43.489131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:43.489186 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:43.489486 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:43.489486 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:43.489532 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.489603 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.490043 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":340,"thread_start_us":66,"threads_started":1}
I20260929 03:22:43.490314 2816 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46757
I20260929 03:22:43.490429 2816 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.490621 2816 heartbeater.cc:511] Master 127.27.68.126:46757 requested a full tablet report, sending...
I20260929 03:22:43.491062 2818 heartbeater.cc:347] Connected to a master server at 127.27.68.125:36901
I20260929 03:22:43.491115 2818 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.491101 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.478384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:59627 (local address 127.27.68.124:38451)
0929 03:22:43.478456 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.478458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.478752 (+ 294us) server_negotiation.cc:407] Connection header received
0929 03:22:43.479591 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.479592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.479601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.479664 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.480371 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.480888 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.481937 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.482163 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.482266 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.485984 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.485986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.485987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.485996 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.488582 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.489053 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.489054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.489054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.489107 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.489287 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.489287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.489288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.489400 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.489459 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.490970 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.491040 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:22:43.491180 2818 heartbeater.cc:511] Master 127.27.68.125:36901 requested a full tablet report, sending...
I20260929 03:22:43.491410 1573 ts_manager.cc:194] Re-registered known tserver with Master: f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875)
I20260929 03:22:43.491617 2815 heartbeater.cc:347] Connected to a master server at 127.27.68.124:38451
I20260929 03:22:43.491689 2815 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:43.491765 2815 heartbeater.cc:511] Master 127.27.68.124:38451 requested a full tablet report, sending...
I20260929 03:22:43.491901 1573 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:33743
I20260929 03:22:43.492048 2362 ts_manager.cc:194] Registered new tserver with Master: f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875)
I20260929 03:22:43.492436 2174 ts_manager.cc:194] Registered new tserver with Master: f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875)
I20260929 03:22:43.495700 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.485151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.27.68.126:46757)
0929 03:22:43.485279 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.485281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.486131 (+ 850us) server_negotiation.cc:407] Connection header received
0929 03:22:43.487111 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.487112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.487127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.487247 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.488057 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.488852 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.489831 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.490124 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.490193 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.491764 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.491765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.491766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.491772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.493916 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.494260 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.494261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.494261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.494321 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.494536 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.494537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.494537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.494683 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.495454 (+ 771us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.495501 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.495597 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260929 03:22:43.497510 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.484898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.27.68.124:38451)
0929 03:22:43.485496 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.485498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.486326 (+ 828us) server_negotiation.cc:407] Connection header received
0929 03:22:43.486348 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.486349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.486363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.486421 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.487285 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.488124 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.492846 (+ 4722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.493146 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.493257 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.494547 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.494548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.494549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.494556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.496050 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.496443 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.496444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.496444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.496501 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.496700 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.496700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.496701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.496848 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.496952 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.497368 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.497447 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":60,"threads_started":1}
I20260929 03:22:43.497503 1573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:43.499405 1573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:43.502790 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.485194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.27.68.125:36901)
0929 03:22:43.490297 (+ 5103us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.490299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.493016 (+ 2717us) server_negotiation.cc:407] Connection header received
0929 03:22:43.493116 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.493117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.493127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.493162 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.493823 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.494593 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.495618 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.495922 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.495995 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.499858 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.499859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.499859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.499866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.501555 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.501962 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.501963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.501964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.502023 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.502250 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.502251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.502251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.502411 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.502505 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.502605 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.502686 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5084,"thread_start_us":69,"threads_started":1}
I20260929 03:22:43.507702 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.504194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.27.68.66:42355)
0929 03:22:43.504703 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.504716 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.504726 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:43.504765 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.504767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.504785 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.504889 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.505093 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505782 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.506711 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.507332 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.507431 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.507510 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.507562 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.507603 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":80,"threads_started":1}
I20260929 03:22:43.507699 2133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.503854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42355 (local address 127.0.0.1:54698)
0929 03:22:43.504022 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.504032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.504105 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.504885 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.504886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.504887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.504914 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.504915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505853 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.505854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.506604 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.506607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.507432 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.507432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.507493 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.507497 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.507546 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.507593 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3}
I20260929 03:22:43.508572 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.503867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37789 (local address 127.0.0.1:59156)
0929 03:22:43.504146 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.504156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.504247 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.504438 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.504439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.504441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.504496 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.504497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505786 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.505787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.507009 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.507012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.508300 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.508301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.508350 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.508354 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.508426 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.508471 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":38,"thread_start_us":93,"threads_started":1}
I20260929 03:22:43.508675 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.503964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.27.68.67:38875)
0929 03:22:43.504261 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.504265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.504523 (+ 258us) server_negotiation.cc:407] Connection header received
0929 03:22:43.504596 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.504598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.504616 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.504701 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.504854 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505871 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.507402 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.508329 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.508403 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.508468 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.508552 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.508592 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20260929 03:22:43.508695 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.503854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:38875 (local address 127.0.0.1:49580)
0929 03:22:43.504457 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.504467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.504528 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.504706 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.504707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.504709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.504772 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.504773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505931 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.505932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.507318 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.507322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.508414 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.508415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.508475 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.508477 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.508536 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.508574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":768,"thread_start_us":89,"threads_started":1}
I20260929 03:22:43.508715 2612 tablet_service.cc:1467] Processing CreateTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 (DEFAULT_TABLE table=test-table [id=9b15de424fd24b8686b86e695a55a93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:43.508903 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.503965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.27.68.65:37789)
0929 03:22:43.504159 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.504165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.504240 (+ 75us) server_negotiation.cc:407] Connection header received
0929 03:22:43.504317 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.504319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.504335 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.504431 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.504573 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.505687 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.507101 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.508116 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.508196 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.508269 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.508767 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.508820 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260929 03:22:43.509192 2612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1022fdf1b3c4d2286ebc131b48087b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:43.509554 2475 tablet_service.cc:1467] Processing CreateTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 (DEFAULT_TABLE table=test-table [id=9b15de424fd24b8686b86e695a55a93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:43.509815 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1022fdf1b3c4d2286ebc131b48087b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:43.512002 2749 tablet_service.cc:1467] Processing CreateTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 (DEFAULT_TABLE table=test-table [id=9b15de424fd24b8686b86e695a55a93a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:43.512384 2749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1022fdf1b3c4d2286ebc131b48087b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:43.512456 2842 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Bootstrap starting.
I20260929 03:22:43.512964 2843 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Bootstrap starting.
I20260929 03:22:43.513135 2842 tablet_bootstrap.cc:654] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:43.513491 2842 log.cc:824] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:43.513666 2843 tablet_bootstrap.cc:654] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:43.513939 2843 log.cc:824] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:43.514334 2842 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: No bootstrap required, opened a new log
I20260929 03:22:43.514420 2842 ts_tablet_manager.cc:1397] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:43.514688 2843 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: No bootstrap required, opened a new log
I20260929 03:22:43.514760 2843 ts_tablet_manager.cc:1397] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:43.515857 2846 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Bootstrap starting.
I20260929 03:22:43.516588 2846 tablet_bootstrap.cc:654] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:43.516728 2843 raft_consensus.cc:348] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.516880 2843 raft_consensus.cc:374] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:43.516935 2843 raft_consensus.cc:729] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ece50a719974528abcafc4a287fe9c2, State: Initialized, Role: FOLLOWER
I20260929 03:22:43.517032 2846 log.cc:824] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:43.517058 2843 consensus_queue.cc:260] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.517014 2842 raft_consensus.cc:348] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.517185 2842 raft_consensus.cc:374] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:43.517231 2842 raft_consensus.cc:729] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d33d0f582f4413958e5fb2dcc6428e, State: Initialized, Role: FOLLOWER
I20260929 03:22:43.517316 2843 ts_tablet_manager.cc:1428] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:43.517359 2842 consensus_queue.cc:260] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.517462 2542 heartbeater.cc:503] Master 127.27.68.126:46757 was elected leader, sending a full tablet report...
I20260929 03:22:43.517904 2679 heartbeater.cc:503] Master 127.27.68.126:46757 was elected leader, sending a full tablet report...
I20260929 03:22:43.517606 2842 ts_tablet_manager.cc:1428] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:22:43.519150 2846 tablet_bootstrap.cc:492] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: No bootstrap required, opened a new log
I20260929 03:22:43.519316 2846 ts_tablet_manager.cc:1397] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:22:43.521286 2846 raft_consensus.cc:348] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.521430 2846 raft_consensus.cc:374] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:43.521452 2846 raft_consensus.cc:729] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82b909bbac847c5bbca02862f1e3bf2, State: Initialized, Role: FOLLOWER
I20260929 03:22:43.521589 2846 consensus_queue.cc:260] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.521900 2846 ts_tablet_manager.cc:1428] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:43.521937 2816 heartbeater.cc:503] Master 127.27.68.126:46757 was elected leader, sending a full tablet report...
I20260929 03:22:43.547767 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.545372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.27.68.124:38451)
0929 03:22:43.545430 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.545432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.545442 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:43.545471 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.545472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.545491 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.545555 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.545676 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.546158 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.547014 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.547489 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.547552 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.547616 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.547674 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.547704 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:22:43.547770 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.545300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56540)
0929 03:22:43.545365 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.545378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.545425 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.545569 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.545570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.545571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.545618 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.545620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.546224 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.546225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.546935 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.546937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.547563 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.547564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.547616 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.547619 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.547668 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.547710 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:22:43.548197 2372 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 03:22:43.549072 2401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab2e724b3f546d9b8c643c36bdc5581. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.549175 2401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: LEARNER
I20260929 03:22:43.569016 2849 raft_consensus.cc:482] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:43.569203 2849 raft_consensus.cc:504] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.569608 2849 leader_election.cc:288] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875), 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:43.573666 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.569932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:58041 (local address 127.27.68.67:38875)
0929 03:22:43.569983 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.569985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.569997 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:22:43.570030 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.570031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.570041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.570103 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.570326 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.571077 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.572702 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.573253 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.573345 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.573356 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.573537 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.573588 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:22:43.573700 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.569904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:56091 (local address 127.27.68.65:37789)
0929 03:22:43.569951 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.569953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.570010 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:22:43.570079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.570080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.570090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.570139 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:43.570326 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.571077 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.572702 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.573293 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.573353 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.573361 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.573588 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.573639 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:22:43.573693 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.569828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:38875 (local address 127.27.68.66:58041)
0929 03:22:43.569901 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.569914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.569968 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.570137 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.570138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.570141 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.570244 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.570245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.571174 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.571174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.572591 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.572595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.573367 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.573368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.573462 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.573466 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.573508 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.573565 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":75}
I20260929 03:22:43.573693 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.569828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37789 (local address 127.27.68.66:56091)
0929 03:22:43.569956 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:43.569965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:43.570015 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:43.570152 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:43.570152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:43.570154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:43.570244 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.570245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.571173 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.571174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.572591 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:43.572595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.573397 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.573398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:43.573485 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.573488 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:43.573566 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:22:43.573605 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":169}
I20260929 03:22:43.574088 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1022fdf1b3c4d2286ebc131b48087b4" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ece50a719974528abcafc4a287fe9c2" is_pre_election: true
I20260929 03:22:43.574142 2769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1022fdf1b3c4d2286ebc131b48087b4" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82b909bbac847c5bbca02862f1e3bf2" is_pre_election: true
I20260929 03:22:43.574280 2495 raft_consensus.cc:2454] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 0.
I20260929 03:22:43.574343 2769 raft_consensus.cc:2454] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 0.
I20260929 03:22:43.574530 2568 leader_election.cc:302] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ece50a719974528abcafc4a287fe9c2, 73d33d0f582f4413958e5fb2dcc6428e; no voters:
I20260929 03:22:43.574671 2849 raft_consensus.cc:2781] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:43.574728 2849 raft_consensus.cc:482] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:43.574754 2849 raft_consensus.cc:3037] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:22:43.575102 2681 tablet.cc:2369] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:43.575569 2849 raft_consensus.cc:504] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.575737 2849 leader_election.cc:288] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 election: Requested vote from peers f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875), 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:43.575865 2769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1022fdf1b3c4d2286ebc131b48087b4" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82b909bbac847c5bbca02862f1e3bf2"
I20260929 03:22:43.575954 2769 raft_consensus.cc:3037] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:43.575912 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1022fdf1b3c4d2286ebc131b48087b4" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ece50a719974528abcafc4a287fe9c2"
I20260929 03:22:43.576020 2495 raft_consensus.cc:3037] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:43.576639 2769 raft_consensus.cc:2454] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 1.
I20260929 03:22:43.576838 2567 leader_election.cc:302] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d33d0f582f4413958e5fb2dcc6428e, f82b909bbac847c5bbca02862f1e3bf2; no voters:
I20260929 03:22:43.576889 2495 raft_consensus.cc:2454] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 1.
I20260929 03:22:43.576936 2849 raft_consensus.cc:2781] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:43.577091 2849 raft_consensus.cc:686] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 1 LEADER]: Becoming Leader. State: Replica: 73d33d0f582f4413958e5fb2dcc6428e, State: Running, Role: LEADER
I20260929 03:22:43.577199 2849 consensus_queue.cc:237] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:43.578349 1573 catalog_manager.cc:5495] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66). New cstate: current_term: 1 leader_uuid: "73d33d0f582f4413958e5fb2dcc6428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:43.578848 2325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: NON_VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: true } } }
I20260929 03:22:43.578948 2325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.579906 2326 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: attempting to promote NON_VOTER cfeb4ce56f364dff8ab2f996b722b71c to VOTER
I20260929 03:22:43.580056 2326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:43.580479 2184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 03:22:43.580488 2372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 LEARNER]: Refusing update from remote peer 0ab2e724b3f546d9b8c643c36bdc5581: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 03:22:43.580698 2858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 03:22:43.580823 2325 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 03:22:43.581426 2326 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.581624 2372 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582212 2325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582216 2326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582319 2325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.582373 2326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.582417 2325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582463 2325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581 [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:43.582379 2184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582525 2401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.582607 2401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:22:43.583416 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:43.583590 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:22:43.602651 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.600431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.27.68.66:42355)
0929 03:22:43.600494 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.600496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.600507 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:43.600571 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.600572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.600598 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.600691 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:43.600797 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.601258 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.601955 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.602136 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.602187 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.602321 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:43.602426 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:43.602490 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.602519 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.602537 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:22:43.604566 2495 raft_consensus.cc:1261] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Refusing update from remote peer 73d33d0f582f4413958e5fb2dcc6428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:43.604574 2769 raft_consensus.cc:1261] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Refusing update from remote peer 73d33d0f582f4413958e5fb2dcc6428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:43.604984 2855 consensus_queue.cc:1035] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:43.605132 2849 consensus_queue.cc:1035] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:43.608814 2862 mvcc.cc:205] Tried to move back new op lower bound from 7334511262121545728 to 7334511262012825600. Current Snapshot: MvccSnapshot[applied={T|T < 7334511262121545728 or (T in {7334511262121545728})}]
I20260929 03:22:43.608939 2863 mvcc.cc:205] Tried to move back new op lower bound from 7334511262121545728 to 7334511262012825600. Current Snapshot: MvccSnapshot[applied={T|T < 7334511262121545728}]
I20260929 03:22:43.609642 2864 mvcc.cc:205] Tried to move back new op lower bound from 7334511262121545728 to 7334511262012825600. Current Snapshot: MvccSnapshot[applied={T|T < 7334511262121545728}]
I20260929 03:22:43.610850 1574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:43.610952 1574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:43.612558 1574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ab2e724b3f546d9b8c643c36bdc5581: Sending DeleteTablet for 3 replicas of tablet b1022fdf1b3c4d2286ebc131b48087b4
I20260929 03:22:43.612993 2612 tablet_service.cc:1514] Processing DeleteTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:43 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:54698
I20260929 03:22:43.613087 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:43 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:59156
I20260929 03:22:43.613229 2872 tablet_replica.cc:330] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: stopping tablet replica
I20260929 03:22:43.613324 2872 raft_consensus.cc:2229] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:43.613415 2872 raft_consensus.cc:2258] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:43.613473 2749 tablet_service.cc:1514] Processing DeleteTablet for tablet b1022fdf1b3c4d2286ebc131b48087b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:43 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:49580
I20260929 03:22:43.614003 2872 ts_tablet_manager.cc:1905] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:43.614030 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 1542
I20260929 03:22:43.616077 2872 ts_tablet_manager.cc:1918] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:43.616161 2872 log.cc:1196] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/b1022fdf1b3c4d2286ebc131b48087b4
I20260929 03:22:43.616441 2872 ts_tablet_manager.cc:1939] T b1022fdf1b3c4d2286ebc131b48087b4 P 73d33d0f582f4413958e5fb2dcc6428e: Deleting consensus metadata
I20260929 03:22:43.621124 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.613491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.27.68.124:38451)
0929 03:22:43.613535 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.613537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.613728 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:22:43.613833 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.613834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.613845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.613876 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.614727 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.615479 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.616805 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.617025 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.617058 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.618436 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.618437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.618437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.618444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.619945 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.620327 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.620329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.620329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.620389 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.620592 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.620593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.620593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.620714 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.620823 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.620932 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.621015 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:22:43.621367 2567 connection.cc:535] server connection from 127.0.0.1:54698 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:22:43.621618 2873 tablet_replica.cc:330] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: stopping tablet replica
I20260929 03:22:43.621754 2873 raft_consensus.cc:2229] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:43.621886 2873 raft_consensus.cc:2258] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:43.622714 2875 tablet_replica.cc:330] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: stopping tablet replica
I20260929 03:22:43.622851 2875 raft_consensus.cc:2229] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:43.622987 2875 raft_consensus.cc:2258] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:43.623296 2873 ts_tablet_manager.cc:1905] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:43.623458 2875 ts_tablet_manager.cc:1905] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:43.625018 2875 ts_tablet_manager.cc:1918] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:43.625018 2873 ts_tablet_manager.cc:1918] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:43.625102 2873 log.cc:1196] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/b1022fdf1b3c4d2286ebc131b48087b4
I20260929 03:22:43.625102 2875 log.cc:1196] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/b1022fdf1b3c4d2286ebc131b48087b4
I20260929 03:22:43.625387 2875 ts_tablet_manager.cc:1939] T b1022fdf1b3c4d2286ebc131b48087b4 P f82b909bbac847c5bbca02862f1e3bf2: Deleting consensus metadata
I20260929 03:22:43.625387 2873 ts_tablet_manager.cc:1939] T b1022fdf1b3c4d2286ebc131b48087b4 P 5ece50a719974528abcafc4a287fe9c2: Deleting consensus metadata
W20260929 03:22:43.625769 2431 connection.cc:441] server connection from 127.0.0.1:59156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59156 (request call id 1) could send its response
W20260929 03:22:43.625895 2704 connection.cc:441] server connection from 127.0.0.1:49580 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49580 (request call id 1) could send its response
I20260929 03:22:43.629774 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.622790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.27.68.124:38451)
0929 03:22:43.622831 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.622833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.623048 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:22:43.623147 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.623148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.623159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.623189 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.623921 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.624396 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.625638 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.625867 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.625924 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.627279 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.627280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.627280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.627287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.628662 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.629008 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.629009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.629009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.629066 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.629270 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.629271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.629271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.629389 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.629494 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.629603 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.629677 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:43.636343 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:43.623066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.27.68.125:36901)
0929 03:22:43.623111 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:43.623113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:43.629827 (+ 6714us) server_negotiation.cc:407] Connection header received
0929 03:22:43.630160 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:43.630161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:43.630173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:43.630229 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:43.630698 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.631360 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.632147 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:43.632478 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:43.632540 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:43.633866 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:43.633868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:43.633868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:43.633876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:43.635149 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.635551 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.635552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.635553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.635611 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:43.635819 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:43.635820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:43.635820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:43.635937 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:43.636036 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:43.636154 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:22:43.636237 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:44.609469 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:44.609190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:45061)
0929 03:22:44.609321 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:44.609410 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260929 03:22:44.609876 2816 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:44.610137 2884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:44.609865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:39351)
0929 03:22:44.609978 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:44.610067 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
I20260929 03:22:44.610280 2883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:44.609846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:41953)
0929 03:22:44.610161 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:44.610233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":56,"threads_started":1}
W20260929 03:22:44.610437 2679 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
W20260929 03:22:44.610600 2542 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.113103 2886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ab2e724b3f546d9b8c643c36bdc5581)
I20260929 03:22:45.113097 2885 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0ab2e724b3f546d9b8c643c36bdc5581)
I20260929 03:22:45.113229 2886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:45.113270 2885 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:45.113536 2885 leader_election.cc:288] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125:36901)
I20260929 03:22:45.113754 2886 leader_election.cc:288] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124:38451)
I20260929 03:22:45.113986 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42014)
0929 03:22:45.113895 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.113947 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":75,"threads_started":1}
I20260929 03:22:45.114070 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42016)
0929 03:22:45.113777 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.114015 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":48,"threads_started":1}
W20260929 03:22:45.114174 2159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
W20260929 03:22:45.114159 2348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.114606 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.114521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42034)
0929 03:22:45.114556 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.114580 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 03:22:45.114678 2159 leader_election.cc:334] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.114763 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.114639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42050)
0929 03:22:45.114697 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.114733 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 03:22:45.114873 2348 leader_election.cc:334] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.117051 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.27.68.124:38451)
0929 03:22:45.113994 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:45.113996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:45.114005 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:45.114030 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:45.114031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:45.114042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:45.114274 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:45.114316 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.115100 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.116310 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.116694 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.116785 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:45.116796 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:45.116944 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:22:45.116985 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260929 03:22:45.117064 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:38451 (local address 127.0.0.1:56570)
0929 03:22:45.113835 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.113846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:45.113907 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:45.114136 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:45.114137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:45.114139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:45.114208 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:45.114210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.115180 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.115181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:45.116217 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:45.116219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.116789 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.116790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:45.116858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:45.116862 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:45.116936 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:22:45.116985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":41,"threads_started":1}
I20260929 03:22:45.117358 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.27.68.125:36901)
0929 03:22:45.114403 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:45.114405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:45.114411 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:22:45.114430 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:45.114430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:45.114438 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:45.114936 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:45.114967 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.115720 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.116636 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.117023 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.117172 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:45.117184 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:45.117266 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:22:45.117315 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":39,"threads_started":1}
I20260929 03:22:45.117357 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.113837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:36901 (local address 127.0.0.1:40396)
0929 03:22:45.114053 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.114062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:45.114116 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:45.114836 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:45.114837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:45.114840 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:45.114880 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:45.114882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.115796 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.115797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:45.116542 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:45.116544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:45.117124 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:45.117126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:45.117189 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:45.117193 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:45.117245 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:22:45.117284 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":51,"threads_started":1}
I20260929 03:22:45.117573 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ae8f015a6a474197bc50393cfb4faf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" is_pre_election: true
I20260929 03:22:45.117580 2184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51ae8f015a6a474197bc50393cfb4faf" is_pre_election: true
I20260929 03:22:45.117678 2372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51ae8f015a6a474197bc50393cfb4faf in term 1.
I20260929 03:22:45.117678 2184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfeb4ce56f364dff8ab2f996b722b71c in term 1.
I20260929 03:22:45.117841 2160 leader_election.cc:302] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51ae8f015a6a474197bc50393cfb4faf, cfeb4ce56f364dff8ab2f996b722b71c; no voters: 0ab2e724b3f546d9b8c643c36bdc5581
I20260929 03:22:45.117836 2348 leader_election.cc:302] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51ae8f015a6a474197bc50393cfb4faf, cfeb4ce56f364dff8ab2f996b722b71c; no voters: 0ab2e724b3f546d9b8c643c36bdc5581
I20260929 03:22:45.117951 2886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:22:45.117992 2886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0ab2e724b3f546d9b8c643c36bdc5581)
I20260929 03:22:45.117982 2885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:22:45.118023 2886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:22:45.118026 2885 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0ab2e724b3f546d9b8c643c36bdc5581)
I20260929 03:22:45.118057 2885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:22:45.118788 2886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:45.118791 2885 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:45.118923 2885 leader_election.cc:288] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 election: Requested vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125:36901)
I20260929 03:22:45.118937 2886 leader_election.cc:288] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 election: Requested vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), cfeb4ce56f364dff8ab2f996b722b71c (127.27.68.124:38451)
I20260929 03:22:45.119114 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.119012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42068)
0929 03:22:45.119049 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.119080 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:22:45.119138 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "51ae8f015a6a474197bc50393cfb4faf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cfeb4ce56f364dff8ab2f996b722b71c"
I20260929 03:22:45.119262 2372 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 51ae8f015a6a474197bc50393cfb4faf in current term 2: Already voted for candidate cfeb4ce56f364dff8ab2f996b722b71c in this term.
I20260929 03:22:45.119308 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.119057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42080)
0929 03:22:45.119093 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.119126 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 03:22:45.119282 2184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51ae8f015a6a474197bc50393cfb4faf"
I20260929 03:22:45.119364 2184 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cfeb4ce56f364dff8ab2f996b722b71c in current term 2: Already voted for candidate 51ae8f015a6a474197bc50393cfb4faf in this term.
I20260929 03:22:45.119410 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.119323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42086)
0929 03:22:45.119368 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.119390 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:22:45.119519 2159 leader_election.cc:334] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.119549 2159 leader_election.cc:302] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51ae8f015a6a474197bc50393cfb4faf; no voters: 0ab2e724b3f546d9b8c643c36bdc5581, cfeb4ce56f364dff8ab2f996b722b71c
I20260929 03:22:45.119625 2889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.119489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42094)
0929 03:22:45.119577 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.119603 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:22:45.119647 2886 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 03:22:45.119688 2348 leader_election.cc:334] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:45.119733 2348 leader_election.cc:302] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cfeb4ce56f364dff8ab2f996b722b71c; no voters: 0ab2e724b3f546d9b8c643c36bdc5581, 51ae8f015a6a474197bc50393cfb4faf
I20260929 03:22:45.119836 2885 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 03:22:45.610526 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.610292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:45519)
0929 03:22:45.610441 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.610484 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260929 03:22:45.610957 2896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.610716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:51719)
0929 03:22:45.610867 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.610894 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260929 03:22:45.611054 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:45.610863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:55027)
0929 03:22:45.610969 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:45.611005 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":47,"threads_started":1}
I20260929 03:22:46.611316 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.611024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:48061)
0929 03:22:46.611163 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.611236 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260929 03:22:46.611730 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.611447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:52001)
0929 03:22:46.611635 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.611683 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":54,"threads_started":1}
I20260929 03:22:46.612021 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.611758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:43409)
0929 03:22:46.611937 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.611976 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260929 03:22:46.864288 2901 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:46.864408 2901 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:46.864624 2901 leader_election.cc:288] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125:36901)
I20260929 03:22:46.865068 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.864802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42298)
0929 03:22:46.864942 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.865013 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":44,"threads_started":1}
I20260929 03:22:46.865077 2184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51ae8f015a6a474197bc50393cfb4faf" is_pre_election: true
I20260929 03:22:46.865204 2184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfeb4ce56f364dff8ab2f996b722b71c in term 2.
I20260929 03:22:46.865453 2348 leader_election.cc:302] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ae8f015a6a474197bc50393cfb4faf, cfeb4ce56f364dff8ab2f996b722b71c; no voters:
I20260929 03:22:46.865499 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.865404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42302)
0929 03:22:46.865450 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.865477 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:22:46.865545 2348 leader_election.cc:334] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:46.865574 2901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 03:22:46.865617 2901 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:46.865645 2901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:22:46.866381 2901 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:46.866528 2901 leader_election.cc:288] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 election: Requested vote from peers 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757), 51ae8f015a6a474197bc50393cfb4faf (127.27.68.125:36901)
I20260929 03:22:46.866756 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.866641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42310)
0929 03:22:46.866683 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.866732 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:22:46.866834 2184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "51ae8f015a6a474197bc50393cfb4faf"
I20260929 03:22:46.866938 2184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:22:46.867085 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.866983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42318)
0929 03:22:46.867012 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.867059 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 03:22:46.867159 2348 leader_election.cc:334] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
I20260929 03:22:46.867679 2184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfeb4ce56f364dff8ab2f996b722b71c in term 3.
I20260929 03:22:46.867882 2348 leader_election.cc:302] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 51ae8f015a6a474197bc50393cfb4faf, cfeb4ce56f364dff8ab2f996b722b71c; no voters: 0ab2e724b3f546d9b8c643c36bdc5581
I20260929 03:22:46.867962 2901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:22:46.868067 2901 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [term 3 LEADER]: Becoming Leader. State: Replica: cfeb4ce56f364dff8ab2f996b722b71c, State: Running, Role: LEADER
I20260929 03:22:46.868165 2901 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } }
I20260929 03:22:46.868548 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfeb4ce56f364dff8ab2f996b722b71c. Latest consensus state: current_term: 3 leader_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:46.868651 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:46.868836 2906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:46.869264 2906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:46.869321 2906 catalog_manager.cc:1259] Loaded cluster ID: f56b3d4201c04b10a17fc48e60d4415a
I20260929 03:22:46.869374 2906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:46.869565 2906 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:46.869647 2906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Loaded TSK: 0
I20260929 03:22:46.869802 2906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:22:46.877206 2361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:46.878996 2361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:46.880137 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.880011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42338)
0929 03:22:46.880062 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.880099 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:22:46.880111 2184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [term 3 FOLLOWER]: Refusing update from remote peer cfeb4ce56f364dff8ab2f996b722b71c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 03:22:46.880374 2904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 03:22:46.880525 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.880437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.0.0.1:42352)
0929 03:22:46.880478 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.880501 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:22:46.880698 2348 consensus_peers.cc:487] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c -> Peer 0ab2e724b3f546d9b8c643c36bdc5581 (127.27.68.126:46757): Couldn't send request to peer 0ab2e724b3f546d9b8c643c36bdc5581. Status: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:22:46.881110 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfeb4ce56f364dff8ab2f996b722b71c. Latest consensus state: current_term: 3 leader_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:46.881211 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:22:46.881739 2907 mvcc.cc:205] Tried to move back new op lower bound from 7334511275538898944 to 7334511275492708352. Current Snapshot: MvccSnapshot[applied={T|T < 7334511275538898944}]
I20260929 03:22:46.881906 2904 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:46.882002 2904 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:46.882014 2909 mvcc.cc:205] Tried to move back new op lower bound from 7334511275538898944 to 7334511275492708352. Current Snapshot: MvccSnapshot[applied={T|T < 7334511275538898944}]
I20260929 03:22:46.882299 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0ab2e724b3f546d9b8c643c36bdc5581" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46757 } } peers { permanent_uuid: "51ae8f015a6a474197bc50393cfb4faf" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 36901 } attrs { promote: false } } peers { permanent_uuid: "cfeb4ce56f364dff8ab2f996b722b71c" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 38451 } attrs { promote: false } } }
I20260929 03:22:46.882413 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ae8f015a6a474197bc50393cfb4faf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:22:46.887784 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.27.68.66:42355)
0929 03:22:46.884347 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:46.884350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:46.884357 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:22:46.884385 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:46.884386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:46.884399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:46.884468 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:46.884638 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.885403 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.886779 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887403 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887517 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887534 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:46.887650 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:22:46.887706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
I20260929 03:22:46.887776 2902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42355 (local address 127.0.0.1:54714)
0929 03:22:46.884152 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.884164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:46.884219 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:46.884485 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:46.884486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:46.884488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:46.884555 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.884556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.885482 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.885482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.886672 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.886674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887500 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.887565 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887570 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:46.887635 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:22:46.887689 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260929 03:22:46.887979 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:38875 (local address 127.0.0.1:49590)
0929 03:22:46.884271 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.884282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:46.884340 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:46.884574 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:46.884575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:46.884576 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:46.884610 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.884611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.886099 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.886100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.886933 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.886935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887693 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.887797 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887801 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:46.887849 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:22:46.887894 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260929 03:22:46.888084 2612 tablet_service.cc:1467] Processing CreateTablet for tablet 74e35e1bad67436dab80aebd12bdcaac (DEFAULT_TABLE table=test-table [id=7137bdfd0d2343419aea55435d29b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:46.888193 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.27.68.67:38875)
0929 03:22:46.884272 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:46.884274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:46.884429 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:22:46.884455 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:46.884456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:46.884466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:46.884531 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:46.884675 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.886014 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887051 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887584 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887643 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887654 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:46.888074 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 03:22:46.888120 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260929 03:22:46.888240 2612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e35e1bad67436dab80aebd12bdcaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:46.888909 2749 tablet_service.cc:1467] Processing CreateTablet for tablet 74e35e1bad67436dab80aebd12bdcaac (DEFAULT_TABLE table=test-table [id=7137bdfd0d2343419aea55435d29b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:46.888971 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37789 (local address 127.0.0.1:59168)
0929 03:22:46.884318 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.884328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:46.884375 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:46.884559 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:46.884560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:46.884562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:46.884609 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.884611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.885654 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.885655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.886691 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:46.886693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887759 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:46.887826 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887831 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:46.888843 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0929 03:22:46.888894 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260929 03:22:46.889030 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.884149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.27.68.65:37789)
0929 03:22:46.884424 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:46.884426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:46.884433 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:22:46.884474 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:46.884475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:46.884485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:46.884555 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:46.884891 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.885585 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887016 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:46.887644 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:46.887715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:46.887727 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:46.888898 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0929 03:22:46.888949 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":67,"threads_started":1}
I20260929 03:22:46.889046 2749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e35e1bad67436dab80aebd12bdcaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:46.889645 2918 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Bootstrap starting.
I20260929 03:22:46.889773 2475 tablet_service.cc:1467] Processing CreateTablet for tablet 74e35e1bad67436dab80aebd12bdcaac (DEFAULT_TABLE table=test-table [id=7137bdfd0d2343419aea55435d29b352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:46.889890 2475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e35e1bad67436dab80aebd12bdcaac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:46.890074 2918 tablet_bootstrap.cc:654] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:46.890627 2918 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: No bootstrap required, opened a new log
I20260929 03:22:46.890686 2918 ts_tablet_manager.cc:1397] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:46.890825 2918 raft_consensus.cc:348] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.890892 2918 raft_consensus.cc:374] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:46.890910 2918 raft_consensus.cc:729] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d33d0f582f4413958e5fb2dcc6428e, State: Initialized, Role: FOLLOWER
I20260929 03:22:46.890959 2918 consensus_queue.cc:260] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.891183 2922 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: Bootstrap starting.
I20260929 03:22:46.891475 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.891335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:34403)
0929 03:22:46.891414 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.891444 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 03:22:46.891673 2922 tablet_bootstrap.cc:654] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:46.891122 2918 ts_tablet_manager.cc:1428] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:46.892035 2678 heartbeater.cc:503] Master 127.27.68.124:38451 was elected leader, sending a full tablet report...
I20260929 03:22:46.892130 2920 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Bootstrap starting.
I20260929 03:22:46.892516 2920 tablet_bootstrap.cc:654] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:46.892544 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.892251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:51149)
0929 03:22:46.892484 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.892515 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260929 03:22:46.893134 2922 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: No bootstrap required, opened a new log
I20260929 03:22:46.893149 2920 tablet_bootstrap.cc:492] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: No bootstrap required, opened a new log
I20260929 03:22:46.893182 2922 ts_tablet_manager.cc:1397] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:22:46.893219 2920 ts_tablet_manager.cc:1397] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:46.893347 2922 raft_consensus.cc:348] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.893405 2922 raft_consensus.cc:374] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:46.893419 2922 raft_consensus.cc:729] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ece50a719974528abcafc4a287fe9c2, State: Initialized, Role: FOLLOWER
I20260929 03:22:46.893369 2920 raft_consensus.cc:348] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.893450 2920 raft_consensus.cc:374] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:46.893482 2920 raft_consensus.cc:729] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82b909bbac847c5bbca02862f1e3bf2, State: Initialized, Role: FOLLOWER
I20260929 03:22:46.893496 2922 consensus_queue.cc:260] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.893532 2920 consensus_queue.cc:260] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:46.893708 2922 ts_tablet_manager.cc:1428] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:46.893741 2920 ts_tablet_manager.cc:1428] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:46.893975 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.893848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:36839)
0929 03:22:46.893909 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.893940 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:22:46.894140 2541 heartbeater.cc:503] Master 127.27.68.124:38451 was elected leader, sending a full tablet report...
I20260929 03:22:46.894166 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.893982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:42175)
0929 03:22:46.894109 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.894135 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 03:22:46.894362 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.894275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:40957)
0929 03:22:46.894311 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.894332 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:22:46.894517 2815 heartbeater.cc:503] Master 127.27.68.124:38451 was elected leader, sending a full tablet report...
I20260929 03:22:46.894840 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:46.894760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:47769)
0929 03:22:46.894794 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:46.894816 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 03:22:46.922046 2544 tablet.cc:2369] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:22:46.980517 2819 tablet.cc:2369] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:47.017542 2925 raft_consensus.cc:482] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:47.017666 2925 raft_consensus.cc:504] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:47.018074 2925 leader_election.cc:288] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875), 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355)
I20260929 03:22:47.019062 2921 raft_consensus.cc:482] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:47.019138 2921 raft_consensus.cc:504] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:47.019302 2921 leader_election.cc:288] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875), 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:47.019479 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ece50a719974528abcafc4a287fe9c2" is_pre_election: true
I20260929 03:22:47.019510 2769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82b909bbac847c5bbca02862f1e3bf2" is_pre_election: true
I20260929 03:22:47.019563 2495 raft_consensus.cc:2454] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 0.
I20260929 03:22:47.019587 2769 raft_consensus.cc:2454] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 0.
I20260929 03:22:47.019698 2568 leader_election.cc:302] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ece50a719974528abcafc4a287fe9c2, 73d33d0f582f4413958e5fb2dcc6428e; no voters:
I20260929 03:22:47.019788 2921 raft_consensus.cc:2781] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:47.019826 2921 raft_consensus.cc:482] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:47.019850 2921 raft_consensus.cc:3037] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:47.020721 2921 raft_consensus.cc:504] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:47.020870 2921 leader_election.cc:288] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 election: Requested vote from peers f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875), 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789)
I20260929 03:22:47.021029 2495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ece50a719974528abcafc4a287fe9c2"
I20260929 03:22:47.021021 2769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "73d33d0f582f4413958e5fb2dcc6428e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82b909bbac847c5bbca02862f1e3bf2"
I20260929 03:22:47.021081 2495 raft_consensus.cc:3037] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:47.021081 2769 raft_consensus.cc:3037] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:47.021384 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.018280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:38875 (local address 127.27.68.65:57021)
0929 03:22:47.018340 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.018354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:47.018433 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:47.018589 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:47.018590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:47.018592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:47.018658 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:47.018660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.019488 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.019489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:47.020459 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:47.020462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.021136 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.021137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:47.021198 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:47.021212 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:22:47.021256 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:22:47.021296 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:22:47.021592 2915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.018377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:57021 (local address 127.27.68.67:38875)
0929 03:22:47.018421 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:47.018424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:47.018433 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:47.018498 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:47.018499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:47.018507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:47.018584 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:47.018740 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.019399 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.020566 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.021030 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.021095 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:47.021105 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:47.021469 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 03:22:47.021510 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:22:47.021713 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.018384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38413 (local address 127.27.68.66:42355)
0929 03:22:47.018440 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:47.018443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:47.018641 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:22:47.018730 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:47.018731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:47.018741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:47.018818 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:47.018950 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.019644 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.020860 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.021359 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.021423 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:47.021433 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:47.021594 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 03:22:47.021636 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:22:47.021821 2769 raft_consensus.cc:2454] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 1.
I20260929 03:22:47.021821 2495 raft_consensus.cc:2454] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d33d0f582f4413958e5fb2dcc6428e in term 1.
I20260929 03:22:47.021950 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.018280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42355 (local address 127.27.68.65:38413)
0929 03:22:47.018559 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.018568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:47.018649 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:47.018820 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:47.018821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:47.018822 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:47.018869 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:47.018872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.019728 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.019729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:47.020754 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:47.020757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.021465 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.021466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:47.021517 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:47.021520 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:22:47.021814 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 03:22:47.021879 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":83,"thread_start_us":127,"threads_started":1}
I20260929 03:22:47.021981 2567 leader_election.cc:302] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d33d0f582f4413958e5fb2dcc6428e, f82b909bbac847c5bbca02862f1e3bf2; no voters:
I20260929 03:22:47.022055 2921 raft_consensus.cc:2781] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:47.022087 2769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "5ece50a719974528abcafc4a287fe9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f82b909bbac847c5bbca02862f1e3bf2" is_pre_election: true
I20260929 03:22:47.022173 2769 raft_consensus.cc:2379] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ece50a719974528abcafc4a287fe9c2 in current term 1: Already voted for candidate 73d33d0f582f4413958e5fb2dcc6428e in this term.
I20260929 03:22:47.022208 2921 raft_consensus.cc:686] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 1 LEADER]: Becoming Leader. State: Replica: 73d33d0f582f4413958e5fb2dcc6428e, State: Running, Role: LEADER
I20260929 03:22:47.022267 2921 consensus_queue.cc:237] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } }
I20260929 03:22:47.022764 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.022665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:38517)
0929 03:22:47.022703 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.022734 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:22:47.022825 2632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e35e1bad67436dab80aebd12bdcaac" candidate_uuid: "5ece50a719974528abcafc4a287fe9c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d33d0f582f4413958e5fb2dcc6428e" is_pre_election: true
I20260929 03:22:47.023245 2428 leader_election.cc:302] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ece50a719974528abcafc4a287fe9c2; no voters: 73d33d0f582f4413958e5fb2dcc6428e, f82b909bbac847c5bbca02862f1e3bf2
I20260929 03:22:47.023253 2362 catalog_manager.cc:5495] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66). New cstate: current_term: 1 leader_uuid: "73d33d0f582f4413958e5fb2dcc6428e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d33d0f582f4413958e5fb2dcc6428e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:47.023397 2925 raft_consensus.cc:2726] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 03:22:47.035595 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.033532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.27.68.66:42355)
0929 03:22:47.033583 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:47.033585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:47.033599 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:22:47.033640 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:47.033640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:47.033648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:47.033691 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:47.033784 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.034255 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.034961 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:47.035141 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:47.035191 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:47.035389 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:47.035418 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:47.035459 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:47.035514 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:22:47.035535 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:22:47.036528 2495 raft_consensus.cc:1261] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Refusing update from remote peer 73d33d0f582f4413958e5fb2dcc6428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:47.036716 2769 raft_consensus.cc:1261] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Refusing update from remote peer 73d33d0f582f4413958e5fb2dcc6428e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:47.036770 2928 consensus_queue.cc:1035] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ece50a719974528abcafc4a287fe9c2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:47.036991 2928 consensus_queue.cc:1035] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f82b909bbac847c5bbca02862f1e3bf2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:47.037709 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.037503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:54393)
0929 03:22:47.037629 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.037662 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 03:22:47.038116 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.038011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:58531)
0929 03:22:47.038056 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.038082 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 03:22:47.038497 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.038372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:48827)
0929 03:22:47.038437 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.038464 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 03:22:47.038889 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.038753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.66:43101)
0929 03:22:47.038838 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.038863 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 03:22:47.038996 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.038812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.65:41239)
0929 03:22:47.038935 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.038963 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 03:22:47.039660 2898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.039558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46757 (local address 127.27.68.67:57341)
0929 03:22:47.039594 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.039629 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:22:47.040930 2361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:47.041021 2361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:47.042356 2361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfeb4ce56f364dff8ab2f996b722b71c: Sending DeleteTablet for 3 replicas of tablet 74e35e1bad67436dab80aebd12bdcaac
I20260929 03:22:47.042631 2749 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e35e1bad67436dab80aebd12bdcaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:47 UTC) from {username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:49590
I20260929 03:22:47.042659 2475 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e35e1bad67436dab80aebd12bdcaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:47 UTC) from {username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:59168
I20260929 03:22:47.042658 2612 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e35e1bad67436dab80aebd12bdcaac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:47 UTC) from {username='oryx', principal='oryx/127.27.68.124@KRBTEST.COM'} at 127.0.0.1:54714
I20260929 03:22:47.042883 2942 tablet_replica.cc:330] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: stopping tablet replica
I20260929 03:22:47.042951 2942 raft_consensus.cc:2229] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:47.043018 2942 raft_consensus.cc:2258] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:47.043120 2943 tablet_replica.cc:330] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: stopping tablet replica
I20260929 03:22:47.043179 2943 raft_consensus.cc:2229] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:47.043236 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2322
I20260929 03:22:47.043303 2944 tablet_replica.cc:330] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2: stopping tablet replica
I20260929 03:22:47.043365 2944 raft_consensus.cc:2229] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:47.043417 2944 raft_consensus.cc:2258] T 74e35e1bad67436dab80aebd12bdcaac P 5ece50a719974528abcafc4a287fe9c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:47.043524 2943 raft_consensus.cc:2258] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:47.044039 2943 ts_tablet_manager.cc:1905] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:47.045778 2943 ts_tablet_manager.cc:1918] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:47.045857 2943 log.cc:1196] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/74e35e1bad67436dab80aebd12bdcaac
I20260929 03:22:47.046098 2943 ts_tablet_manager.cc:1939] T 74e35e1bad67436dab80aebd12bdcaac P 73d33d0f582f4413958e5fb2dcc6428e: Deleting consensus metadata
I20260929 03:22:47.046461 2346 catalog_manager.cc:4841] TS 73d33d0f582f4413958e5fb2dcc6428e (127.27.68.66:42355): tablet 74e35e1bad67436dab80aebd12bdcaac (table test-table [id=7137bdfd0d2343419aea55435d29b352]) successfully deleted
I20260929 03:22:47.046728 2942 ts_tablet_manager.cc:1905] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:47.048003 2942 ts_tablet_manager.cc:1918] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:47.048084 2942 log.cc:1196] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/74e35e1bad67436dab80aebd12bdcaac
I20260929 03:22:47.048302 2942 ts_tablet_manager.cc:1939] T 74e35e1bad67436dab80aebd12bdcaac P f82b909bbac847c5bbca02862f1e3bf2: Deleting consensus metadata
I20260929 03:22:47.048609 2348 catalog_manager.cc:4841] TS f82b909bbac847c5bbca02862f1e3bf2 (127.27.68.67:38875): tablet 74e35e1bad67436dab80aebd12bdcaac (table test-table [id=7137bdfd0d2343419aea55435d29b352]) successfully deleted
I20260929 03:22:47.050312 2914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:47.050182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:37789 (local address 127.0.0.1:59174)
0929 03:22:47.050242 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:47.050272 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:37789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:22:47.050462 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2549
W20260929 03:22:47.050494 2349 catalog_manager.cc:4584] TS 5ece50a719974528abcafc4a287fe9c2 (127.27.68.65:37789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74e35e1bad67436dab80aebd12bdcaac: Network error: Client connection negotiation failed: client connection to 127.27.68.65:37789: connect: Connection refused (error 111)
I20260929 03:22:47.057021 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2685
I20260929 03:22:47.061825 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2038
I20260929 03:22:47.066781 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2229
2026-09-29T03:22:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6398 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:47 dist-test-slave-fzwk krb5kdc[2947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:47 dist-test-slave-fzwk krb5kdc[2947](info): set up 2 sockets
Sep 29 03:22:47 dist-test-slave-fzwk krb5kdc[2947](info): commencing operation
krb5kdc: starting...
W20260929 03:22:48.104141 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab.
Entry for principal oryx/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab.
I20260929 03:22:48.222344 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35295
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab
--principal=oryx/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:48.305105 2960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:48.305346 2960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:48.305372 2960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:48.306568 2960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:48.306607 2960 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:22:48.306622 2960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:48.306635 2960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:48.306648 2960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:48.306659 2960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:48.308247 2960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35295
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.126.keytab
--principal=oryx/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:48.308496 2960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:48.308683 2960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:48.313745 2968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.313740 2966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.313771 2965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:48.313941 2960 server_base.cc:1034] running on GCE node
I20260929 03:22:48.314292 2960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:48.314616 2960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:48.315783 2960 hybrid_clock.cc:648] HybridClock initialized: now 1790652168315776 us; error 49 us; skew 500 ppm
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:48.324162 2960 init.cc:377] Logged in from keytab as oryx/127.27.68.126@KRBTEST.COM (short username oryx)
I20260929 03:22:48.324662 2960 webserver.cc:460] Webserver started at http://127.27.68.126:40955/ using document root <none> and password file <none>
I20260929 03:22:48.324928 2960 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:48.325004 2960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:48.325166 2960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:48.326277 2960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "62dd49fe870d4c3ca3bf5b7b96b624af"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "724aed2b733ec02c35523262b92a4ed8"
server_key_iv: "6ccdba8c29f01afcd9c58a108f821330"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.326658 2960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "62dd49fe870d4c3ca3bf5b7b96b624af"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "724aed2b733ec02c35523262b92a4ed8"
server_key_iv: "6ccdba8c29f01afcd9c58a108f821330"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.328193 2960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:48.329041 2975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.329241 2960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:48.329348 2960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "62dd49fe870d4c3ca3bf5b7b96b624af"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "724aed2b733ec02c35523262b92a4ed8"
server_key_iv: "6ccdba8c29f01afcd9c58a108f821330"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.329435 2960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:48.340013 2960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:48.341852 2960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:48.342018 2960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:48.346587 2960 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35295
I20260929 03:22:48.346657 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35295 every 8 connection(s)
I20260929 03:22:48.347015 2960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:48.347522 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 2960
I20260929 03:22:48.347604 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:48.347815 27921 external_mini_cluster.cc:1428] Setting key 5860c7015914ea061f781848930064f2
I20260929 03:22:48.347929 3038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:48.350800 3038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Bootstrap starting.
I20260929 03:22:48.351529 3038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:48.351833 3038 log.cc:824] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:48.352602 3038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: No bootstrap required, opened a new log
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:48.354833 3038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } }
I20260929 03:22:48.354975 3038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:48.355000 3038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd49fe870d4c3ca3bf5b7b96b624af, State: Initialized, Role: FOLLOWER
I20260929 03:22:48.355155 3038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } }
I20260929 03:22:48.355283 3038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:48.355327 3038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:48.355382 3038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:48.356081 3038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } }
I20260929 03:22:48.356173 3038 leader_election.cc:302] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62dd49fe870d4c3ca3bf5b7b96b624af; no voters:
I20260929 03:22:48.356374 3038 leader_election.cc:288] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:48.356405 3043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:48.356509 3043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [term 1 LEADER]: Becoming Leader. State: Replica: 62dd49fe870d4c3ca3bf5b7b96b624af, State: Running, Role: LEADER
I20260929 03:22:48.356590 3043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } }
I20260929 03:22:48.356635 3038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:48.356921 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } } }
I20260929 03:22:48.356993 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:48.357136 3043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd49fe870d4c3ca3bf5b7b96b624af. Latest consensus state: current_term: 1 leader_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd49fe870d4c3ca3bf5b7b96b624af" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35295 } } }
I20260929 03:22:48.357206 3043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:48.357775 3048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:48.358480 3048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:48.359117 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.348343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.27.68.126:35295)
0929 03:22:48.348559 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.348567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.348594 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:22:48.349029 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.349036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.349088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.349231 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.349870 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.350792 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.351645 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.352110 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.352249 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.354831 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.354835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.354835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.354846 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.356909 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.357368 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.357371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.357371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.357474 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.357746 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.357747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.357747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.358070 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.358702 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.358782 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.358919 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260929 03:22:48.360221 3048 catalog_manager.cc:1347] Generated new cluster ID: d91b458051544857a6462ebbb6c2fd56
I20260929 03:22:48.360273 3048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:48.374083 3048 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:48.374646 3048 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:48.380780 3048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Generated new TSK 0
I20260929 03:22:48.381033 3048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
I20260929 03:22:48.410967 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35295
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:48.497730 3064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:48.498008 3064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:48.498054 3064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:48.499372 3064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:48.499446 3064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:48.499481 3064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:48.499518 3064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:48.501194 3064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:48.501525 3064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:48.501802 3064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:48.507103 3072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.507182 3070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.507104 3069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:48.507303 3064 server_base.cc:1034] running on GCE node
I20260929 03:22:48.507622 3064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:48.507957 3064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:48.509116 3064 hybrid_clock.cc:648] HybridClock initialized: now 1790652168509099 us; error 50 us; skew 500 ppm
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:48.511960 3064 init.cc:377] Logged in from keytab as oryx/127.27.68.65@KRBTEST.COM (short username oryx)
I20260929 03:22:48.512421 3064 webserver.cc:460] Webserver started at http://127.27.68.65:41263/ using document root <none> and password file <none>
I20260929 03:22:48.512624 3064 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:48.512667 3064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:48.512766 3064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:48.513868 3064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "e4463e944fd14b969d383f6092aede1a"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "a9055580a444432e710f7502a60b6924"
server_key_iv: "60d840318900b704b20cb3ef7497028b"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.514207 3064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "e4463e944fd14b969d383f6092aede1a"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "a9055580a444432e710f7502a60b6924"
server_key_iv: "60d840318900b704b20cb3ef7497028b"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.515836 3064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:48.516736 3079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.516927 3064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:48.517016 3064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "e4463e944fd14b969d383f6092aede1a"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "a9055580a444432e710f7502a60b6924"
server_key_iv: "60d840318900b704b20cb3ef7497028b"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.517115 3064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:48.534106 3064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:48.536155 3064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:48.536360 3064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:48.536902 3064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:48.536969 3064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.537015 3064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:48.537045 3064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:48.544322 3064 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:40763
I20260929 03:22:48.544433 3194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:40763 every 8 connection(s)
I20260929 03:22:48.544752 3064 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:48.546653 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3064
I20260929 03:22:48.546777 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:48.546856 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.537489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:59929 (local address 127.27.68.126:35295)
0929 03:22:48.537540 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.537542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.537820 (+ 278us) server_negotiation.cc:407] Connection header received
0929 03:22:48.538397 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.538398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.538411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.538472 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.539547 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.540072 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.541132 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.541351 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.541430 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.543816 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.543818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.543818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.543827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.545584 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.545967 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.545968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.545968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.546019 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.546231 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.546232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.546232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.546334 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.546384 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.546646 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.546743 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:48.546998 27921 external_mini_cluster.cc:1428] Setting key 832f7faa8e6e69045b255f288c21430e
I20260929 03:22:48.547048 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.537503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.65:59929)
0929 03:22:48.537689 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.537733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.538295 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.538606 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.538612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.539029 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.539434 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.539440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.540177 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.540178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.541021 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.541023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.541455 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.541456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.541610 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.542190 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.542205 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.543735 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.545660 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.545665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.545667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.545893 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.546078 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.546078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.546079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.546176 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.546388 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.546392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.546510 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.546596 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.546765 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260929 03:22:48.547654 3195 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35295
I20260929 03:22:48.547777 3195 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:48.547976 3195 heartbeater.cc:511] Master 127.27.68.126:35295 requested a full tablet report, sending...
I20260929 03:22:48.548525 2992 ts_manager.cc:194] Registered new tserver with Master: e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763)
I20260929 03:22:48.549340 2992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:59929
WARNING: no policy specified for oryx/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:22:48.557464 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.548735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:49727 (local address 127.27.68.126:35295)
0929 03:22:48.548769 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.548771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.551404 (+ 2633us) server_negotiation.cc:407] Connection header received
0929 03:22:48.551433 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.551434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.551447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.551514 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.551875 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.552688 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.553376 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.553702 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.553781 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.554719 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.554721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.554721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.554729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.556337 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.556720 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.556721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.556721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.556772 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.556934 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.556934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.556935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.557076 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.557137 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.557291 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.557368 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:22:48.557466 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.548687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.65:49727)
0929 03:22:48.551327 (+ 2640us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.551339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.551397 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.551540 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.551541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.551747 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.551798 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.551800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.552786 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.552786 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.553294 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.553295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.553782 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.553783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.553833 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.554330 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.554334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.554621 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.556418 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.556418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.556419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.556645 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.556828 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.556829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.556829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.556885 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.557126 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.557127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.557167 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.557244 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.557361 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
I20260929 03:22:48.573926 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35295
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:48.655094 3204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:48.655328 3204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:48.655354 3204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:48.656656 3204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:48.656698 3204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:48.656720 3204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:48.656772 3204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:48.658465 3204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:48.658736 3204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:48.658978 3204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:48.663966 3209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.663966 3212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.663993 3210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:48.664350 3204 server_base.cc:1034] running on GCE node
I20260929 03:22:48.664564 3204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:48.664762 3204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:48.665954 3204 hybrid_clock.cc:648] HybridClock initialized: now 1790652168665929 us; error 43 us; skew 500 ppm
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:48.668576 3204 init.cc:377] Logged in from keytab as oryx/127.27.68.66@KRBTEST.COM (short username oryx)
I20260929 03:22:48.669092 3204 webserver.cc:460] Webserver started at http://127.27.68.66:42703/ using document root <none> and password file <none>
I20260929 03:22:48.669344 3204 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:48.669417 3204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:48.669579 3204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:48.670686 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "0d1065acf86647f398a3486877ba078d"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "76ac04f79709743a51cebd3e0baad101"
server_key_iv: "b08c01ba36308cc8b6715402f57be04f"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.671094 3204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "0d1065acf86647f398a3486877ba078d"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "76ac04f79709743a51cebd3e0baad101"
server_key_iv: "b08c01ba36308cc8b6715402f57be04f"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.672590 3204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:48.673374 3219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.673559 3204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:48.673667 3204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "0d1065acf86647f398a3486877ba078d"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "76ac04f79709743a51cebd3e0baad101"
server_key_iv: "b08c01ba36308cc8b6715402f57be04f"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.673758 3204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:48.684653 3204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:48.686760 3204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:48.686962 3204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:48.687674 3204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:48.687741 3204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.687788 3204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:48.687819 3204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:48.696180 3204 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:36637
I20260929 03:22:48.696480 3334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:36637 every 8 connection(s)
I20260929 03:22:48.696633 3204 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:48.698495 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.688106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51667 (local address 127.27.68.126:35295)
0929 03:22:48.688177 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.688180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.688377 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:22:48.689054 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.689055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.689073 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.689168 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.689945 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.690794 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.691977 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.692361 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.692470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.695277 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.695279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.695279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.695288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.697066 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.697470 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.697472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.697472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.697537 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.697796 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.697797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.697797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.697942 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.698014 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.698312 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.698401 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:22:48.698685 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.688083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.66:51667)
0929 03:22:48.688233 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.688277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.688930 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.689300 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.689304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.689619 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.689831 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.689836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.690920 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.690921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.691891 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.691894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.692512 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.692513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.692669 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.693283 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.693307 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.695132 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.697149 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.697154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.697156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.697382 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.697600 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.697601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.697602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.697721 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.698014 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.698022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.698167 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.698281 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.698449 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260929 03:22:48.699241 3335 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35295
I20260929 03:22:48.699335 3335 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:48.699522 3335 heartbeater.cc:511] Master 127.27.68.126:35295 requested a full tablet report, sending...
I20260929 03:22:48.699512 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3204
I20260929 03:22:48.699626 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:48.699872 27921 external_mini_cluster.cc:1428] Setting key 5c862eddbd235e107be497142180fb2b
I20260929 03:22:48.699951 2992 ts_manager.cc:194] Registered new tserver with Master: 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637)
I20260929 03:22:48.700331 2992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:51667
I20260929 03:22:48.707106 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.700801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51681 (local address 127.27.68.126:35295)
0929 03:22:48.700837 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.700839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.700879 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:22:48.700946 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.700946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.700957 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.701010 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.701374 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.702122 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.703194 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.703449 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.703550 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.704499 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.704501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.704501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.704508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.705875 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.706237 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.706252 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.706252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.706308 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.706512 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.706513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.706513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.706658 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.706732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.706913 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.707004 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:22:48.707106 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.700694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.66:51681)
0929 03:22:48.700812 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.700824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.700894 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.701056 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.701057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.701239 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.701290 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.701291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.702228 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.702228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.703101 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.703104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.703553 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.703554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.703628 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.704117 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.704121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.704405 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.705965 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.705975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.705975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.706154 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.706374 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.706375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.706376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.706436 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.706734 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.706735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.706781 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.706885 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.707002 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for oryx/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
I20260929 03:22:48.726024 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35295
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:48.809803 3344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:48.810082 3344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:48.810128 3344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:48.811442 3344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:48.811522 3344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:48.811563 3344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:22:48.811605 3344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:48.813226 3344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.68.126:35295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:48.813547 3344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:48.813797 3344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:48.818878 3350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.818892 3352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:48.818903 3349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:48.819291 3344 server_base.cc:1034] running on GCE node
I20260929 03:22:48.819535 3344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:48.819746 3344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:48.820888 3344 hybrid_clock.cc:648] HybridClock initialized: now 1790652168820878 us; error 30 us; skew 500 ppm
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:48.823750 3344 init.cc:377] Logged in from keytab as oryx/127.27.68.67@KRBTEST.COM (short username oryx)
I20260929 03:22:48.824303 3344 webserver.cc:460] Webserver started at http://127.27.68.67:41523/ using document root <none> and password file <none>
I20260929 03:22:48.824589 3344 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:48.824640 3344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:48.824800 3344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:48.825915 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "3ad2ec4d9f1444779b91587747d507ed"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "30743edea3eea26d853aeaf3de205ae6"
server_key_iv: "a2a4d07fdba19c25a07350d41a1616cc"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.826283 3344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "3ad2ec4d9f1444779b91587747d507ed"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "30743edea3eea26d853aeaf3de205ae6"
server_key_iv: "a2a4d07fdba19c25a07350d41a1616cc"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.827795 3344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:48.828605 3359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.828811 3344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:48.828922 3344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "3ad2ec4d9f1444779b91587747d507ed"
format_stamp: "Formatted at 2026-09-29 03:22:48 on dist-test-slave-fzwk"
server_key: "30743edea3eea26d853aeaf3de205ae6"
server_key_iv: "a2a4d07fdba19c25a07350d41a1616cc"
server_key_version: "encryptionkey@0"
I20260929 03:22:48.829016 3344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:48.840672 3344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:48.843196 3344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:48.843432 3344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:48.844048 3344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:48.844111 3344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:48.844154 3344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:48.844177 3344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:48.851835 3344 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:44233
I20260929 03:22:48.851919 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:44233 every 8 connection(s)
I20260929 03:22:48.852288 3344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:48.854264 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.844487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:42461 (local address 127.27.68.126:35295)
0929 03:22:48.844552 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.844555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.844792 (+ 237us) server_negotiation.cc:407] Connection header received
0929 03:22:48.845274 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.845276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.845291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.845384 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.846056 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.846928 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.848196 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.848551 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.848629 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.851414 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.851416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.851417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.851425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.853136 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.853499 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.853500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.853501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.853554 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.853760 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.853761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.853761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.853864 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.853916 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.854135 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.854196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:22:48.854501 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.844478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.67:42461)
0929 03:22:48.844620 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.844661 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.845179 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.845443 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.845447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.845780 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.845975 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.845981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.847079 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.847080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.848118 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.848120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.848669 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.848670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.848812 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.849389 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.849399 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.851316 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.853210 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.853214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.853216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.853428 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.853610 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.853611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.853611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.853706 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.853919 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.853922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.854033 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.854114 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.854244 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260929 03:22:48.855020 3477 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35295
I20260929 03:22:48.855099 3477 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:48.855312 3477 heartbeater.cc:511] Master 127.27.68.126:35295 requested a full tablet report, sending...
I20260929 03:22:48.855721 2990 ts_manager.cc:194] Registered new tserver with Master: 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:48.856233 2990 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:42461
I20260929 03:22:48.861400 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3344
I20260929 03:22:48.861505 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:48.861696 27921 external_mini_cluster.cc:1428] Setting key 1a5e14f489c48847af10c0d9f40a70cc
I20260929 03:22:48.861967 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.856165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:43275 (local address 127.27.68.126:35295)
0929 03:22:48.856203 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.856205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.856328 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:22:48.856425 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.856426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.856437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.856499 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.856860 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.857398 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.858152 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.858353 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.858434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.859532 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.859533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.859533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.859540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.860735 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.861087 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.861089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.861089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.861151 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.861385 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.861386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.861386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.861538 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.861624 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.861795 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.861876 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:22:48.861972 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.856101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.27.68.67:43275)
0929 03:22:48.856264 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.856273 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.856343 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.856500 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.856501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.856680 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:48.856779 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.856781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.857483 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.857484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.858064 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.858065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.858448 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.858449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.858511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.858993 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:48.858998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:48.859431 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:48.860821 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.860822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.860822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.861000 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.861227 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:48.861228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:48.861228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:48.861301 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:48.861629 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:48.861630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:48.861681 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.861763 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.861878 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260929 03:22:48.862818 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:22:48.884658 3041 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:48.878789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.27.68.126:35295)
0929 03:22:48.878852 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.878855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.878866 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:48.878928 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.878930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.878947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.879015 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.879702 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.880212 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.881040 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.881239 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.881352 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.884555 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38682: BlockingRecv error: recv got EOF from 127.0.0.1:38682 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 03:22:48 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:48.892467 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.885507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.27.68.126:35295)
0929 03:22:48.885574 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.885576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.885660 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:22:48.885742 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.885743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.885755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.885826 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:48.886313 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.886794 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.887561 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.887753 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.887826 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.890009 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:48.890011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:48.890012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:48.890020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:48.891317 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.891652 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.891653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.891653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.891717 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:48.891912 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:48.891913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:48.891913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:48.892036 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:48.892145 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.892270 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.892359 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:22:48.894200 2992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:48.895954 2992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:48.902920 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.898393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40763 (local address 127.0.0.1:50054)
0929 03:22:48.898543 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.898613 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.898683 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.898871 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.898873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.898876 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.898945 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.898947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900523 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.900524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.901641 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.901645 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.902572 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.902573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.902655 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.902670 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.902753 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.902812 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260929 03:22:48.902961 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.898497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.27.68.65:40763)
0929 03:22:48.898679 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.898685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.898697 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:22:48.898771 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.898774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.898790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.898865 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.899015 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900373 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.901736 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.902480 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.902559 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.902637 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.902782 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.902835 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260929 03:22:48.903208 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.898408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:36637 (local address 127.0.0.1:53224)
0929 03:22:48.898666 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.898678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.898729 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.899156 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.899157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.899160 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.899241 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.899243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900548 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.900548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.901897 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.901899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.902833 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.902834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.902898 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.902903 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.902933 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.903139 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":137,"thread_start_us":81,"threads_started":1}
I20260929 03:22:48.903590 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.899061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.27.68.67:44233)
0929 03:22:48.899444 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.899449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.899458 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:48.899494 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.899496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.899512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.899585 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.899690 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900446 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.902214 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.902867 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.902930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.903007 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.903421 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.903467 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":74,"threads_started":1}
I20260929 03:22:48.903721 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.898794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.27.68.66:36637)
0929 03:22:48.898953 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.898958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.898971 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:22:48.899011 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.899013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.899028 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.899116 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.899328 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900464 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.901982 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.902714 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.903374 (+ 660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.903462 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.903495 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.903536 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260929 03:22:48.903774 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.898617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:44233 (local address 127.0.0.1:57390)
0929 03:22:48.899218 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.899278 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.899325 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.899581 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.899582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.899584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.899636 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.899638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.900527 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.900529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.902126 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.902128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.903045 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.903046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.903112 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.903116 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.903161 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.903200 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":155,"thread_start_us":92,"threads_started":1}
I20260929 03:22:48.903767 3129 tablet_service.cc:1467] Processing CreateTablet for tablet dfbfcc79059e44158b21b838e73e2de3 (DEFAULT_TABLE table=test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:48.904186 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbfcc79059e44158b21b838e73e2de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:48.904489 3410 tablet_service.cc:1467] Processing CreateTablet for tablet dfbfcc79059e44158b21b838e73e2de3 (DEFAULT_TABLE table=test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:48.904800 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbfcc79059e44158b21b838e73e2de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:48.907577 3505 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Bootstrap starting.
I20260929 03:22:48.908077 3269 tablet_service.cc:1467] Processing CreateTablet for tablet dfbfcc79059e44158b21b838e73e2de3 (DEFAULT_TABLE table=test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:48.908234 3505 tablet_bootstrap.cc:654] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:48.908452 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfbfcc79059e44158b21b838e73e2de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:48.908565 3505 log.cc:824] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:48.909344 3505 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: No bootstrap required, opened a new log
I20260929 03:22:48.909420 3505 ts_tablet_manager.cc:1397] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:48.911597 3505 raft_consensus.cc:348] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.911798 3505 raft_consensus.cc:374] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:48.911837 3505 raft_consensus.cc:729] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Initialized, Role: FOLLOWER
I20260929 03:22:48.912040 3505 consensus_queue.cc:260] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.912170 3509 raft_consensus.cc:482] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:48.912283 3509 raft_consensus.cc:504] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.912336 3505 ts_tablet_manager.cc:1428] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:48.912462 3195 heartbeater.cc:503] Master 127.27.68.126:35295 was elected leader, sending a full tablet report...
I20260929 03:22:48.912808 3511 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Bootstrap starting.
I20260929 03:22:48.913501 3511 tablet_bootstrap.cc:654] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:48.913803 3511 log.cc:824] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:48.913947 3509 leader_election.cc:288] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:48.914134 3506 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Bootstrap starting.
I20260929 03:22:48.914909 3511 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: No bootstrap required, opened a new log
I20260929 03:22:48.914981 3511 ts_tablet_manager.cc:1397] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:48.916160 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.912926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40917 (local address 127.27.68.66:36637)
0929 03:22:48.912961 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.912963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.913062 (+ 99us) server_negotiation.cc:407] Connection header received
0929 03:22:48.913159 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.913160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.913169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.913197 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.913403 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.914070 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.915415 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.915785 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.915882 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.915892 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.916062 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.916097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:22:48.916183 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.912795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:36637 (local address 127.27.68.65:40917)
0929 03:22:48.913003 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.913013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.913086 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.913258 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.913258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.913260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.913328 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.913330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.914292 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.914293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.915320 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.915323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.915885 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.915886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.915963 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.915969 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.916050 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.916092 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260929 03:22:48.916546 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.912795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:44233 (local address 127.27.68.65:40663)
0929 03:22:48.912903 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.912915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.912953 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.913144 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.913145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.913148 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.913217 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.913219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.914162 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.914163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.915350 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.915354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.916277 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.916278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.916367 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.916371 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.916422 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.916468 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 03:22:48.916497 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d" is_pre_election: true
I20260929 03:22:48.916812 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.912885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40663 (local address 127.27.68.67:44233)
0929 03:22:48.912992 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.912994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.913002 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:48.913026 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.913027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.913035 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.913281 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.913313 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.914055 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.915462 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.916011 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.916087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.916095 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.916693 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.916735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 03:22:48.916909 3081 leader_election.cc:341] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637): Illegal state: must be running to vote when last-logged opid is not known
I20260929 03:22:48.917140 3511 raft_consensus.cc:348] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.917284 3511 raft_consensus.cc:374] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:48.917327 3511 raft_consensus.cc:729] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Initialized, Role: FOLLOWER
I20260929 03:22:48.917450 3511 consensus_queue.cc:260] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.917762 3511 ts_tablet_manager.cc:1428] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:22:48.917804 3335 heartbeater.cc:503] Master 127.27.68.126:35295 was elected leader, sending a full tablet report...
I20260929 03:22:48.917120 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed" is_pre_election: true
I20260929 03:22:48.918099 3506 tablet_bootstrap.cc:654] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Neither blocks nor log segments found. Creating new log.
W20260929 03:22:48.918442 3082 leader_election.cc:341] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 03:22:48.918504 3506 log.cc:824] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:48.918499 3082 leader_election.cc:302] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4463e944fd14b969d383f6092aede1a; no voters: 0d1065acf86647f398a3486877ba078d, 3ad2ec4d9f1444779b91587747d507ed
I20260929 03:22:48.918746 3509 raft_consensus.cc:2726] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 03:22:48.919363 3506 tablet_bootstrap.cc:492] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: No bootstrap required, opened a new log
I20260929 03:22:48.919437 3506 ts_tablet_manager.cc:1397] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:22:48.920495 3506 raft_consensus.cc:348] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.920603 3506 raft_consensus.cc:374] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:48.920624 3506 raft_consensus.cc:729] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Initialized, Role: FOLLOWER
I20260929 03:22:48.920764 3506 consensus_queue.cc:260] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.921082 3506 ts_tablet_manager.cc:1428] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:22:48.921101 3477 heartbeater.cc:503] Master 127.27.68.126:35295 was elected leader, sending a full tablet report...
W20260929 03:22:48.947352 3336 tablet.cc:2369] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:48.986975 3514 raft_consensus.cc:482] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:48.987116 3514 raft_consensus.cc:504] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.987478 3514 leader_election.cc:288] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:48.990463 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.987712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40763 (local address 127.27.68.66:44731)
0929 03:22:48.987772 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.987786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.987835 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.988003 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.988004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.988006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.988080 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.988081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.988744 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.988746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.989605 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.989608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.990154 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.990155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.990272 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.990276 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.990329 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.990377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:22:48.990502 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.987783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:44731 (local address 127.27.68.65:40763)
0929 03:22:48.987848 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.987850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.987864 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:22:48.987897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.987898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.987909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.987982 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.988153 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.988643 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.989707 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.990068 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.990157 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.990166 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.990380 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.990429 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:48.990972 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "0d1065acf86647f398a3486877ba078d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a" is_pre_election: true
I20260929 03:22:48.991111 3149 raft_consensus.cc:2454] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1065acf86647f398a3486877ba078d in term 0.
I20260929 03:22:48.991381 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.987919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:36669 (local address 127.27.68.67:44233)
0929 03:22:48.987987 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:48.987989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:48.988139 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:22:48.988174 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:48.988175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:48.988186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:48.988260 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:48.988430 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.988921 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.990340 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.990970 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.991064 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.991074 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:48.991262 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 03:22:48.991304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:22:48.991339 3222 leader_election.cc:302] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:48.991475 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:48.987835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:44233 (local address 127.27.68.66:36669)
0929 03:22:48.988047 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:48.988063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:48.988126 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:48.988261 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:48.988262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:48.988264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:48.988360 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.988361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.989018 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.989020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.990238 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:48.990241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:48.991086 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:48.991087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:48.991155 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:48.991159 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:22:48.991250 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:22:48.991292 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":133,"threads_started":1}
I20260929 03:22:48.991513 3514 raft_consensus.cc:2781] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:48.991569 3514 raft_consensus.cc:482] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:48.991595 3514 raft_consensus.cc:3037] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:48.991735 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "0d1065acf86647f398a3486877ba078d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed" is_pre_election: true
I20260929 03:22:48.991900 3430 raft_consensus.cc:2454] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1065acf86647f398a3486877ba078d in term 0.
I20260929 03:22:48.992481 3514 raft_consensus.cc:504] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.992635 3514 leader_election.cc:288] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [CANDIDATE]: Term 1 election: Requested vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:48.992822 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "0d1065acf86647f398a3486877ba078d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed"
I20260929 03:22:48.992856 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfbfcc79059e44158b21b838e73e2de3" candidate_uuid: "0d1065acf86647f398a3486877ba078d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a"
I20260929 03:22:48.992897 3430 raft_consensus.cc:3037] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:48.992950 3149 raft_consensus.cc:3037] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:48.993597 3430 raft_consensus.cc:2454] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1065acf86647f398a3486877ba078d in term 1.
I20260929 03:22:48.993713 3149 raft_consensus.cc:2454] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1065acf86647f398a3486877ba078d in term 1.
I20260929 03:22:48.993794 3222 leader_election.cc:302] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, 3ad2ec4d9f1444779b91587747d507ed; no voters:
I20260929 03:22:48.993886 3514 raft_consensus.cc:2781] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:48.994038 3514 raft_consensus.cc:686] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 1 LEADER]: Becoming Leader. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Running, Role: LEADER
I20260929 03:22:48.994172 3514 consensus_queue.cc:237] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:48.995056 2992 catalog_manager.cc:5495] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d1065acf86647f398a3486877ba078d (127.27.68.66). New cstate: current_term: 1 leader_uuid: "0d1065acf86647f398a3486877ba078d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:49.017616 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.015508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.27.68.66:36637)
0929 03:22:49.015560 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.015562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.015570 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:49.015632 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.015633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.015650 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.015710 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.015829 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.016285 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.016998 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.017175 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.017229 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.017354 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.017451 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.017506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.017544 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.017558 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:22:49.019447 3430 raft_consensus.cc:1261] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Refusing update from remote peer 0d1065acf86647f398a3486877ba078d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.019541 3149 raft_consensus.cc:1261] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Refusing update from remote peer 0d1065acf86647f398a3486877ba078d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.019827 3514 consensus_queue.cc:1035] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.019986 3518 consensus_queue.cc:1035] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.022806 3522 mvcc.cc:205] Tried to move back new op lower bound from 7334511284300775424 to 7334511284200751104. Current Snapshot: MvccSnapshot[applied={T|T < 7334511284300775424}]
I20260929 03:22:49.024523 2990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.024626 2990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.025343 2990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Sending DeleteTablet for 3 replicas of tablet dfbfcc79059e44158b21b838e73e2de3
I20260929 03:22:49.025681 3269 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbfcc79059e44158b21b838e73e2de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53224
I20260929 03:22:49.025692 3129 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbfcc79059e44158b21b838e73e2de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50054
I20260929 03:22:49.025691 3410 tablet_service.cc:1514] Processing DeleteTablet for tablet dfbfcc79059e44158b21b838e73e2de3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:57390
I20260929 03:22:49.025946 3531 tablet_replica.cc:330] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: stopping tablet replica
I20260929 03:22:49.026047 3531 raft_consensus.cc:2229] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.026157 3531 raft_consensus.cc:2258] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.026167 2990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:49.026314 2990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:49.026613 3532 tablet_replica.cc:330] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: stopping tablet replica
I20260929 03:22:49.026631 3531 ts_tablet_manager.cc:1905] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.026712 3532 raft_consensus.cc:2229] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:49.026801 3532 raft_consensus.cc:2258] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.026937 3533 tablet_replica.cc:330] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: stopping tablet replica
I20260929 03:22:49.027019 3533 raft_consensus.cc:2229] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.027468 3533 raft_consensus.cc:2258] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.027832 3269 tablet_service.cc:1467] Processing CreateTablet for tablet c1558ad41f7e473e81b9bb59978eacba (DEFAULT_TABLE table=test-table [id=1f5b7927a30f4fd5a64edc816573fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.027875 3129 tablet_service.cc:1467] Processing CreateTablet for tablet c1558ad41f7e473e81b9bb59978eacba (DEFAULT_TABLE table=test-table [id=1f5b7927a30f4fd5a64edc816573fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.027930 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1558ad41f7e473e81b9bb59978eacba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.027988 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1558ad41f7e473e81b9bb59978eacba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.028597 3531 ts_tablet_manager.cc:1918] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.028662 3531 log.cc:1196] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/dfbfcc79059e44158b21b838e73e2de3
I20260929 03:22:49.028932 3531 ts_tablet_manager.cc:1939] T dfbfcc79059e44158b21b838e73e2de3 P e4463e944fd14b969d383f6092aede1a: Deleting consensus metadata
I20260929 03:22:49.029255 3505 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Bootstrap starting.
I20260929 03:22:49.029347 2978 catalog_manager.cc:4841] TS e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763): tablet dfbfcc79059e44158b21b838e73e2de3 (table test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]) successfully deleted
I20260929 03:22:49.029512 3511 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Bootstrap starting.
I20260929 03:22:49.029652 3505 tablet_bootstrap.cc:654] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.029923 3511 tablet_bootstrap.cc:654] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.029978 3410 tablet_service.cc:1467] Processing CreateTablet for tablet c1558ad41f7e473e81b9bb59978eacba (DEFAULT_TABLE table=test-table [id=1f5b7927a30f4fd5a64edc816573fba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.030061 3533 ts_tablet_manager.cc:1905] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.030085 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1558ad41f7e473e81b9bb59978eacba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.031461 3506 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Bootstrap starting.
I20260929 03:22:49.031893 3506 tablet_bootstrap.cc:654] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.032347 3533 ts_tablet_manager.cc:1918] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.032418 3533 log.cc:1196] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/dfbfcc79059e44158b21b838e73e2de3
I20260929 03:22:49.032424 3511 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: No bootstrap required, opened a new log
I20260929 03:22:49.032470 3511 ts_tablet_manager.cc:1397] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:22:49.032632 3511 raft_consensus.cc:348] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.032790 3511 raft_consensus.cc:374] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.032814 3511 raft_consensus.cc:729] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.032809 3533 ts_tablet_manager.cc:1939] T dfbfcc79059e44158b21b838e73e2de3 P 3ad2ec4d9f1444779b91587747d507ed: Deleting consensus metadata
I20260929 03:22:49.032866 3511 consensus_queue.cc:260] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.032935 3506 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: No bootstrap required, opened a new log
I20260929 03:22:49.032950 3511 ts_tablet_manager.cc:1428] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.032974 3506 ts_tablet_manager.cc:1397] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:49.033116 3506 raft_consensus.cc:348] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.033203 3506 raft_consensus.cc:374] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.033226 3506 raft_consensus.cc:729] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.033262 3506 consensus_queue.cc:260] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.032617 3532 ts_tablet_manager.cc:1905] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.033351 3506 ts_tablet_manager.cc:1428] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.033545 3505 tablet_bootstrap.cc:492] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: No bootstrap required, opened a new log
I20260929 03:22:49.033603 3505 ts_tablet_manager.cc:1397] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 03:22:49.033776 3505 raft_consensus.cc:348] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.033854 3505 raft_consensus.cc:374] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.033880 3505 raft_consensus.cc:729] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.033923 3505 consensus_queue.cc:260] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.034008 3505 ts_tablet_manager.cc:1428] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:49.034152 2978 catalog_manager.cc:4841] TS 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233): tablet dfbfcc79059e44158b21b838e73e2de3 (table test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]) successfully deleted
I20260929 03:22:49.035118 3532 ts_tablet_manager.cc:1918] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.035207 3532 log.cc:1196] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/dfbfcc79059e44158b21b838e73e2de3
I20260929 03:22:49.035506 3532 ts_tablet_manager.cc:1939] T dfbfcc79059e44158b21b838e73e2de3 P 0d1065acf86647f398a3486877ba078d: Deleting consensus metadata
I20260929 03:22:49.035903 2977 catalog_manager.cc:4841] TS 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637): tablet dfbfcc79059e44158b21b838e73e2de3 (table test-table [id=f9643026c50e44c2a4f63cfaa9b15a29]) successfully deleted
W20260929 03:22:49.045456 3196 tablet.cc:2369] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:22:49.103029 3478 tablet.cc:2369] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:22:49.231182 3509 raft_consensus.cc:482] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:49.231344 3509 raft_consensus.cc:504] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.231546 3509 leader_election.cc:288] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:49.231805 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1558ad41f7e473e81b9bb59978eacba" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d" is_pre_election: true
I20260929 03:22:49.231837 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1558ad41f7e473e81b9bb59978eacba" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed" is_pre_election: true
I20260929 03:22:49.231921 3289 raft_consensus.cc:2454] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 0.
I20260929 03:22:49.231966 3430 raft_consensus.cc:2454] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 0.
I20260929 03:22:49.232129 3081 leader_election.cc:302] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:49.232255 3509 raft_consensus.cc:2781] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:49.232293 3509 raft_consensus.cc:482] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:49.232324 3509 raft_consensus.cc:3037] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.233179 3509 raft_consensus.cc:504] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.233361 3509 leader_election.cc:288] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 election: Requested vote from peers 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:49.233530 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1558ad41f7e473e81b9bb59978eacba" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d"
I20260929 03:22:49.233561 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1558ad41f7e473e81b9bb59978eacba" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed"
I20260929 03:22:49.233642 3289 raft_consensus.cc:3037] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.233649 3430 raft_consensus.cc:3037] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.234331 3289 raft_consensus.cc:2454] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 1.
I20260929 03:22:49.234638 3430 raft_consensus.cc:2454] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 1.
I20260929 03:22:49.234828 3082 leader_election.cc:302] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad2ec4d9f1444779b91587747d507ed, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:49.234956 3509 raft_consensus.cc:2781] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:49.235013 3509 raft_consensus.cc:686] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 1 LEADER]: Becoming Leader. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Running, Role: LEADER
I20260929 03:22:49.235080 3509 consensus_queue.cc:237] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.235924 2990 catalog_manager.cc:5495] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a reported cstate change: term changed from 0 to 1, leader changed from <none> to e4463e944fd14b969d383f6092aede1a (127.27.68.65). New cstate: current_term: 1 leader_uuid: "e4463e944fd14b969d383f6092aede1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652168, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.126@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:49.311148 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.304111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.27.68.126:35295)
0929 03:22:49.304169 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.304171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.304321 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:22:49.304400 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.304401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.304412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.304478 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:49.304974 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.305476 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.306209 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.306390 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.306476 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.308623 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:49.308625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:49.308625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:49.308633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:49.309907 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:49.310317 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:49.310318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:49.310318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:49.310377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:49.310571 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:49.310572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:49.310572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:49.310687 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:49.310766 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.310919 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.311010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:22:49.311304 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.304015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35295 (local address 127.0.0.1:38708)
0929 03:22:49.304239 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.304248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.304335 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.304497 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.304498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.304832 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:49.304884 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.304886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.305563 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.305564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.306108 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.306110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.306478 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.306479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.306539 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.307103 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:49.307108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:49.308530 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:49.310008 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:49.310018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:49.310021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:49.310225 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:49.310443 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:49.310444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:49.310444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:49.310505 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:49.310755 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:49.310762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:49.310811 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.310898 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.311084 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":80,"thread_start_us":118,"threads_started":1}
I20260929 03:22:49.313268 2990 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:38708:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:22:49.313465 2990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:49.314967 3129 tablet_service.cc:1467] Processing CreateTablet for tablet 6da5438da6ee41d2875117cf93be71e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dff2f145ba544aaea572883d2f102e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:49.314998 3410 tablet_service.cc:1467] Processing CreateTablet for tablet 6da5438da6ee41d2875117cf93be71e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dff2f145ba544aaea572883d2f102e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:49.315153 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da5438da6ee41d2875117cf93be71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.315156 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da5438da6ee41d2875117cf93be71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.315366 3269 tablet_service.cc:1467] Processing CreateTablet for tablet 6da5438da6ee41d2875117cf93be71e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dff2f145ba544aaea572883d2f102e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:22:49.315506 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da5438da6ee41d2875117cf93be71e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.316607 3505 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: Bootstrap starting.
I20260929 03:22:49.317000 3511 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: Bootstrap starting.
I20260929 03:22:49.317068 3505 tablet_bootstrap.cc:654] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.317301 3511 tablet_bootstrap.cc:654] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.317747 3505 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: No bootstrap required, opened a new log
I20260929 03:22:49.317809 3505 ts_tablet_manager.cc:1397] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:49.317955 3505 raft_consensus.cc:348] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.318040 3505 raft_consensus.cc:374] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.318061 3505 raft_consensus.cc:729] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.318109 3505 consensus_queue.cc:260] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.318219 3505 ts_tablet_manager.cc:1428] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:49.318550 3506 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: Bootstrap starting.
I20260929 03:22:49.318619 3509 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.318688 3511 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: No bootstrap required, opened a new log
I20260929 03:22:49.318708 3509 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.318749 3511 ts_tablet_manager.cc:1397] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:49.318868 3511 raft_consensus.cc:348] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.318958 3511 raft_consensus.cc:374] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.318981 3511 raft_consensus.cc:729] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.318976 3506 tablet_bootstrap.cc:654] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.319034 3511 consensus_queue.cc:260] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.319152 3511 ts_tablet_manager.cc:1428] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.319191 3518 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.319299 3518 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.320242 3506 tablet_bootstrap.cc:492] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: No bootstrap required, opened a new log
I20260929 03:22:49.320299 3506 ts_tablet_manager.cc:1397] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:49.320506 3506 raft_consensus.cc:348] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.320604 3506 raft_consensus.cc:374] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.320619 3506 raft_consensus.cc:729] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.320667 3506 consensus_queue.cc:260] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.320770 3506 ts_tablet_manager.cc:1428] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:49.320914 3516 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.321003 3516 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.357148 3516 raft_consensus.cc:482] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:49.357262 3516 raft_consensus.cc:504] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.357656 3516 leader_election.cc:288] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637)
I20260929 03:22:49.360719 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.357890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45037 (local address 127.27.68.66:36637)
0929 03:22:49.357951 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.357953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.358049 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:22:49.358128 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.358129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.358139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.358193 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:49.358397 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.358878 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.360003 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.360353 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.360448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.360455 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.360608 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.360644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:22:49.360769 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.357825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:36637 (local address 127.27.68.67:45037)
0929 03:22:49.357974 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.357986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.358050 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.358218 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.358219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.358221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:49.358318 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.358320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.358970 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.358971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.359880 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.359883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.360464 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.360465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.360524 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.360545 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.360601 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.360648 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":37,"thread_start_us":85,"threads_started":1}
I20260929 03:22:49.360838 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.357940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:52633 (local address 127.27.68.65:40763)
0929 03:22:49.357997 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.357999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.358099 (+ 100us) server_negotiation.cc:407] Connection header received
0929 03:22:49.358152 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.358153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.358163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.358226 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:22:49.358412 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.358884 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.359945 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.360325 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.360383 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.360390 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.360722 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.360761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:22:49.360993 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da5438da6ee41d2875117cf93be71e2" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d" is_pre_election: true
I20260929 03:22:49.361114 3289 raft_consensus.cc:2454] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 0.
I20260929 03:22:49.361284 3361 leader_election.cc:302] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, 3ad2ec4d9f1444779b91587747d507ed; no voters:
I20260929 03:22:49.361325 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.357828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40763 (local address 127.27.68.67:52633)
0929 03:22:49.358013 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.358023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.358092 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.358240 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.358241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.358243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:22:49.358324 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.358325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.358999 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.358999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.359857 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.359859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.360404 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.360406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.360494 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.360499 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.360553 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.360614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":29,"thread_start_us":87,"threads_started":1}
I20260929 03:22:49.361407 3516 raft_consensus.cc:2781] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:49.361440 3516 raft_consensus.cc:482] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:49.361471 3516 raft_consensus.cc:3037] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.361649 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da5438da6ee41d2875117cf93be71e2" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a" is_pre_election: true
I20260929 03:22:49.361737 3149 raft_consensus.cc:2454] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 0.
I20260929 03:22:49.362504 3516 raft_consensus.cc:504] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.362648 3516 leader_election.cc:288] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 election: Requested vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637)
I20260929 03:22:49.362771 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da5438da6ee41d2875117cf93be71e2" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a"
I20260929 03:22:49.362797 3289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da5438da6ee41d2875117cf93be71e2" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d"
I20260929 03:22:49.362849 3149 raft_consensus.cc:3037] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.362869 3289 raft_consensus.cc:3037] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.363672 3149 raft_consensus.cc:2454] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 1.
I20260929 03:22:49.363672 3289 raft_consensus.cc:2454] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 1.
I20260929 03:22:49.363843 3361 leader_election.cc:302] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, 3ad2ec4d9f1444779b91587747d507ed; no voters:
I20260929 03:22:49.363955 3516 raft_consensus.cc:2781] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:49.364066 3516 raft_consensus.cc:686] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [term 1 LEADER]: Becoming Leader. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Running, Role: LEADER
I20260929 03:22:49.364151 3516 consensus_queue.cc:237] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.364485 3549 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: TxnStatusTablet state changed. Reason: New leader 3ad2ec4d9f1444779b91587747d507ed. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.364606 3549 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:49.364791 3551 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:22:49.364914 3551 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:22:49.364940 2990 catalog_manager.cc:5495] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67). New cstate: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 03:22:49.378363 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.375439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:44233 (local address 127.0.0.1:57398)
0929 03:22:49.375532 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.375546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.375605 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.375776 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.375776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.375778 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:49.375827 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.375829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.376753 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.376754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.377413 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.377416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.377851 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.377852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.377903 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.377920 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.378224 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.378230 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.378258 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.378300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:22:49.378367 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.375496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.27.68.67:44233)
0929 03:22:49.375553 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.375556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.375589 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:22:49.375670 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.375671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.375692 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.375766 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.375894 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.376680 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.377474 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.377777 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.377851 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.378012 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.378158 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.378238 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.378262 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.378283 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:22:49.379550 3149 raft_consensus.cc:1261] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Refusing update from remote peer 3ad2ec4d9f1444779b91587747d507ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.379609 3289 raft_consensus.cc:1261] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Refusing update from remote peer 3ad2ec4d9f1444779b91587747d507ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.379839 3549 consensus_queue.cc:1035] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.379972 3516 consensus_queue.cc:1035] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.380564 3509 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: TxnStatusTablet state changed. Reason: New leader 3ad2ec4d9f1444779b91587747d507ed. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.380671 3509 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.380664 3518 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: TxnStatusTablet state changed. Reason: New leader 3ad2ec4d9f1444779b91587747d507ed. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.380743 3518 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.381686 3523 mvcc.cc:205] Tried to move back new op lower bound from 7334511285776609280 to 7334511285716291584. Current Snapshot: MvccSnapshot[applied={T|T < 7334511285776609280}]
I20260929 03:22:49.381819 3518 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.381896 3518 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 0d1065acf86647f398a3486877ba078d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.382088 3516 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.382175 3516 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:49.381686 3556 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.382368 3556 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P 3ad2ec4d9f1444779b91587747d507ed: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:22:49.383131 3509 tablet_replica.cc:439] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } } }
I20260929 03:22:49.383258 3509 tablet_replica.cc:442] T 6da5438da6ee41d2875117cf93be71e2 P e4463e944fd14b969d383f6092aede1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:22:49.386271 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.383444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.27.68.65:40763)
0929 03:22:49.383479 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.383481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.383489 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:22:49.383529 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.383538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.383562 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.383626 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.383777 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.384591 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.385400 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.385706 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.385784 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.385925 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.386072 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.386150 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.386174 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.386195 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:22:49.392176 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.389036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:37203 (local address 127.27.68.67:44233)
0929 03:22:49.389114 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.389116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.389219 (+ 103us) server_negotiation.cc:407] Connection header received
0929 03:22:49.389266 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.389267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.389278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.389339 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.389499 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.390274 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.391460 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.391650 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.391709 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.391939 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.391967 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.392002 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.392095 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.392114 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:22:49.392575 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.388989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:44233 (local address 127.27.68.65:37203)
0929 03:22:49.389141 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.389152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.389221 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.389365 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.389366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.389367 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:49.389435 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.389437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.390374 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.390376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.391380 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.391383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.391749 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.391750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.391827 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.391848 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.392028 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.392030 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.392089 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.392137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 03:22:49.395148 3430 raft_consensus.cc:1261] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Refusing update from remote peer e4463e944fd14b969d383f6092aede1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.395172 3289 raft_consensus.cc:1261] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Refusing update from remote peer e4463e944fd14b969d383f6092aede1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.395440 3510 consensus_queue.cc:1035] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.395570 3509 consensus_queue.cc:1035] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.397799 3521 mvcc.cc:205] Tried to move back new op lower bound from 7334511285840744448 to 7334511285187641344. Current Snapshot: MvccSnapshot[applied={T|T < 7334511285840744448}]
I20260929 03:22:49.405002 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.402253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:51021 (local address 127.27.68.65:40763)
0929 03:22:49.402291 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.402293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.402453 (+ 160us) server_negotiation.cc:407] Connection header received
0929 03:22:49.402514 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.402515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.402524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.402565 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.402698 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.403185 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.404378 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.404564 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.404618 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.404809 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.404836 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.404866 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.404930 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.404945 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:22:49.405077 3564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.402221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40763 (local address 127.27.68.67:51021)
0929 03:22:49.402385 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.402396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.402455 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:49.402571 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:49.402572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:49.402573 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:22:49.402642 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.402643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.403322 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.403323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.404303 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:49.404306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.404641 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.404642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:49.404713 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.404731 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.404881 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.404882 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:22:49.404928 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:22:49.404970 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260929 03:22:49.414237 2990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.414353 2990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.415004 2990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Sending DeleteTablet for 3 replicas of tablet c1558ad41f7e473e81b9bb59978eacba
I20260929 03:22:49.415297 3129 tablet_service.cc:1514] Processing DeleteTablet for tablet c1558ad41f7e473e81b9bb59978eacba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50054
I20260929 03:22:49.415297 3410 tablet_service.cc:1514] Processing DeleteTablet for tablet c1558ad41f7e473e81b9bb59978eacba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:57390
I20260929 03:22:49.415414 3269 tablet_service.cc:1514] Processing DeleteTablet for tablet c1558ad41f7e473e81b9bb59978eacba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53224
I20260929 03:22:49.415428 3533 tablet_replica.cc:330] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: stopping tablet replica
I20260929 03:22:49.415489 3533 raft_consensus.cc:2229] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.415500 3532 tablet_replica.cc:330] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: stopping tablet replica
I20260929 03:22:49.415529 3533 raft_consensus.cc:2258] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.415557 3532 raft_consensus.cc:2229] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.415599 3532 raft_consensus.cc:2258] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.415603 3531 tablet_replica.cc:330] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: stopping tablet replica
I20260929 03:22:49.415659 3531 raft_consensus.cc:2229] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:49.415755 3531 raft_consensus.cc:2258] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.416098 3533 ts_tablet_manager.cc:1905] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.416100 3531 ts_tablet_manager.cc:1905] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.417801 3533 ts_tablet_manager.cc:1918] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:49.417874 3533 log.cc:1196] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/c1558ad41f7e473e81b9bb59978eacba
I20260929 03:22:49.418052 3531 ts_tablet_manager.cc:1918] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:49.418107 3533 ts_tablet_manager.cc:1939] T c1558ad41f7e473e81b9bb59978eacba P 3ad2ec4d9f1444779b91587747d507ed: Deleting consensus metadata
I20260929 03:22:49.418129 3531 log.cc:1196] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/c1558ad41f7e473e81b9bb59978eacba
I20260929 03:22:49.418363 3531 ts_tablet_manager.cc:1939] T c1558ad41f7e473e81b9bb59978eacba P e4463e944fd14b969d383f6092aede1a: Deleting consensus metadata
I20260929 03:22:49.418470 2978 catalog_manager.cc:4841] TS 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233): tablet c1558ad41f7e473e81b9bb59978eacba (table test-table [id=1f5b7927a30f4fd5a64edc816573fba9]) successfully deleted
I20260929 03:22:49.419147 2978 catalog_manager.cc:4841] TS e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763): tablet c1558ad41f7e473e81b9bb59978eacba (table test-table [id=1f5b7927a30f4fd5a64edc816573fba9]) successfully deleted
I20260929 03:22:49.419533 3532 ts_tablet_manager.cc:1905] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.421218 3532 ts_tablet_manager.cc:1918] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:49.421288 3532 log.cc:1196] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/c1558ad41f7e473e81b9bb59978eacba
I20260929 03:22:49.421517 3532 ts_tablet_manager.cc:1939] T c1558ad41f7e473e81b9bb59978eacba P 0d1065acf86647f398a3486877ba078d: Deleting consensus metadata
I20260929 03:22:49.421888 2977 catalog_manager.cc:4841] TS 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637): tablet c1558ad41f7e473e81b9bb59978eacba (table test-table [id=1f5b7927a30f4fd5a64edc816573fba9]) successfully deleted
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[2947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:22:49.451959 3041 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:49.444638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.27.68.126:35295)
0929 03:22:49.444690 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.444693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.444864 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:22:49.444923 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.444924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.444938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.445006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:49.445775 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.446561 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.447752 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.448068 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.448139 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.451884 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38714: BlockingRecv error: recv got EOF from 127.0.0.1:38714 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[2947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM
I20260929 03:22:49.463303 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.453080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.27.68.126:35295)
0929 03:22:49.453161 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.453163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.453341 (+ 178us) server_negotiation.cc:407] Connection header received
0929 03:22:49.453383 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.453384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.453397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.453464 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:49.454067 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.454935 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.456149 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.456438 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.456518 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.460024 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:49.460026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:49.460026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:49.460036 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:49.461901 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:49.462355 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:49.462356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:49.462357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:49.462419 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:49.462613 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:49.462614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:49.462614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:49.462755 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:49.462833 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.463047 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.463136 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:22:49.465509 2990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:22:49.465695 2990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:49.467180 3129 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 (DEFAULT_TABLE table=test-table [id=15fcddaee646406086d1d231abfbae9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.467325 3269 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 (DEFAULT_TABLE table=test-table [id=15fcddaee646406086d1d231abfbae9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.467379 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9ad2367d7b425e9ba2520e89067839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.467469 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9ad2367d7b425e9ba2520e89067839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.468744 3505 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Bootstrap starting.
I20260929 03:22:49.468885 3511 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Bootstrap starting.
I20260929 03:22:49.469148 3505 tablet_bootstrap.cc:654] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.469254 3511 tablet_bootstrap.cc:654] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.469539 3410 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 (DEFAULT_TABLE table=test-table [id=15fcddaee646406086d1d231abfbae9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.469679 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9ad2367d7b425e9ba2520e89067839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.471084 3506 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Bootstrap starting.
I20260929 03:22:49.471621 3506 tablet_bootstrap.cc:654] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.472078 3511 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: No bootstrap required, opened a new log
I20260929 03:22:49.472134 3511 ts_tablet_manager.cc:1397] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:22:49.472265 3511 raft_consensus.cc:348] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.472339 3511 raft_consensus.cc:374] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.472360 3511 raft_consensus.cc:729] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.472407 3511 consensus_queue.cc:260] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.472500 3511 ts_tablet_manager.cc:1428] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.472574 3506 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: No bootstrap required, opened a new log
I20260929 03:22:49.472627 3506 ts_tablet_manager.cc:1397] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:49.472770 3506 raft_consensus.cc:348] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.472832 3506 raft_consensus.cc:374] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.472851 3506 raft_consensus.cc:729] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.472891 3506 consensus_queue.cc:260] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.472970 3506 ts_tablet_manager.cc:1428] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.475724 3505 tablet_bootstrap.cc:492] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: No bootstrap required, opened a new log
I20260929 03:22:49.475793 3505 ts_tablet_manager.cc:1397] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260929 03:22:49.475945 3505 raft_consensus.cc:348] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.476020 3505 raft_consensus.cc:374] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.476042 3505 raft_consensus.cc:729] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.476083 3505 consensus_queue.cc:260] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.476171 3505 ts_tablet_manager.cc:1428] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:49.685657 3516 raft_consensus.cc:482] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:49.685761 3516 raft_consensus.cc:504] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.685963 3516 leader_election.cc:288] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637)
I20260929 03:22:49.686236 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9ad2367d7b425e9ba2520e89067839" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d" is_pre_election: true
I20260929 03:22:49.686249 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9ad2367d7b425e9ba2520e89067839" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a" is_pre_election: true
I20260929 03:22:49.686337 3288 raft_consensus.cc:2454] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 0.
I20260929 03:22:49.686359 3149 raft_consensus.cc:2454] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 0.
I20260929 03:22:49.686546 3361 leader_election.cc:302] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1065acf86647f398a3486877ba078d, 3ad2ec4d9f1444779b91587747d507ed; no voters:
I20260929 03:22:49.686659 3516 raft_consensus.cc:2781] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:49.686707 3516 raft_consensus.cc:482] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:49.686738 3516 raft_consensus.cc:3037] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.687455 3516 raft_consensus.cc:504] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.687608 3516 leader_election.cc:288] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 election: Requested vote from peers e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763), 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637)
I20260929 03:22:49.687824 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9ad2367d7b425e9ba2520e89067839" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4463e944fd14b969d383f6092aede1a"
I20260929 03:22:49.687836 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9ad2367d7b425e9ba2520e89067839" candidate_uuid: "3ad2ec4d9f1444779b91587747d507ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d"
I20260929 03:22:49.687904 3288 raft_consensus.cc:3037] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.687911 3149 raft_consensus.cc:3037] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.688594 3149 raft_consensus.cc:2454] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 1.
I20260929 03:22:49.688607 3288 raft_consensus.cc:2454] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad2ec4d9f1444779b91587747d507ed in term 1.
I20260929 03:22:49.688805 3362 leader_election.cc:302] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad2ec4d9f1444779b91587747d507ed, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:49.688885 3516 raft_consensus.cc:2781] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:49.688941 3516 raft_consensus.cc:686] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 1 LEADER]: Becoming Leader. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Running, Role: LEADER
I20260929 03:22:49.688997 3516 consensus_queue.cc:237] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.689631 2990 catalog_manager.cc:5495] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67). New cstate: current_term: 1 leader_uuid: "3ad2ec4d9f1444779b91587747d507ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 03:22:49.699856 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.697292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.27.68.67:44233)
0929 03:22:49.697340 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.697342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.697352 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:49.697420 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.697421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.697434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.697510 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.697662 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.698138 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.699083 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.699347 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.699425 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.699586 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.699622 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.699691 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.699748 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.699772 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:22:49.700791 3288 raft_consensus.cc:1261] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Refusing update from remote peer 3ad2ec4d9f1444779b91587747d507ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.700806 3149 raft_consensus.cc:1261] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Refusing update from remote peer 3ad2ec4d9f1444779b91587747d507ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.701010 3516 consensus_queue.cc:1035] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.701109 3556 consensus_queue.cc:1035] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.704207 2992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.704305 2992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.704952 2992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Sending DeleteTablet for 3 replicas of tablet 7c9ad2367d7b425e9ba2520e89067839
I20260929 03:22:49.705236 3410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:57390
I20260929 03:22:49.705340 3533 tablet_replica.cc:330] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: stopping tablet replica
I20260929 03:22:49.705328 3129 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50054
I20260929 03:22:49.705387 3533 raft_consensus.cc:2229] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:49.705433 3531 tablet_replica.cc:330] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: stopping tablet replica
I20260929 03:22:49.705431 3269 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9ad2367d7b425e9ba2520e89067839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53224
I20260929 03:22:49.705490 3531 raft_consensus.cc:2229] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.705494 3533 raft_consensus.cc:2258] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.705513 3532 tablet_replica.cc:330] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: stopping tablet replica
I20260929 03:22:49.705539 3531 raft_consensus.cc:2258] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.705559 3532 raft_consensus.cc:2229] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.705606 3532 raft_consensus.cc:2258] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.705775 2992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:22:49.705861 3531 ts_tablet_manager.cc:1905] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:22:49.705915 2992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:22:49.706004 3533 ts_tablet_manager.cc:1905] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.706321 3532 ts_tablet_manager.cc:1905] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.707515 3531 ts_tablet_manager.cc:1918] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.707574 3531 log.cc:1196] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/7c9ad2367d7b425e9ba2520e89067839
I20260929 03:22:49.707801 3531 ts_tablet_manager.cc:1939] T 7c9ad2367d7b425e9ba2520e89067839 P e4463e944fd14b969d383f6092aede1a: Deleting consensus metadata
I20260929 03:22:49.707885 3533 ts_tablet_manager.cc:1918] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.707907 3532 ts_tablet_manager.cc:1918] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:22:49.707935 3533 log.cc:1196] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/7c9ad2367d7b425e9ba2520e89067839
I20260929 03:22:49.707957 3532 log.cc:1196] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/7c9ad2367d7b425e9ba2520e89067839
I20260929 03:22:49.708145 3533 ts_tablet_manager.cc:1939] T 7c9ad2367d7b425e9ba2520e89067839 P 3ad2ec4d9f1444779b91587747d507ed: Deleting consensus metadata
I20260929 03:22:49.708173 3532 ts_tablet_manager.cc:1939] T 7c9ad2367d7b425e9ba2520e89067839 P 0d1065acf86647f398a3486877ba078d: Deleting consensus metadata
I20260929 03:22:49.708177 2978 catalog_manager.cc:4841] TS e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763): tablet 7c9ad2367d7b425e9ba2520e89067839 (table test-table [id=15fcddaee646406086d1d231abfbae9c]) successfully deleted
I20260929 03:22:49.708493 2977 catalog_manager.cc:4841] TS 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637): tablet 7c9ad2367d7b425e9ba2520e89067839 (table test-table [id=15fcddaee646406086d1d231abfbae9c]) successfully deleted
I20260929 03:22:49.708596 2978 catalog_manager.cc:4841] TS 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233): tablet 7c9ad2367d7b425e9ba2520e89067839 (table test-table [id=15fcddaee646406086d1d231abfbae9c]) successfully deleted
I20260929 03:22:49.709132 3269 tablet_service.cc:1467] Processing CreateTablet for tablet 1573ac6ed7ac425fa183415033aa74ff (DEFAULT_TABLE table=test-table [id=308fce32b54d4b028f6cbfd9ecc5ab4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.709173 3410 tablet_service.cc:1467] Processing CreateTablet for tablet 1573ac6ed7ac425fa183415033aa74ff (DEFAULT_TABLE table=test-table [id=308fce32b54d4b028f6cbfd9ecc5ab4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.709260 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573ac6ed7ac425fa183415033aa74ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.709280 3410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573ac6ed7ac425fa183415033aa74ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.710148 3506 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Bootstrap starting.
I20260929 03:22:49.710412 3511 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Bootstrap starting.
I20260929 03:22:49.710510 3506 tablet_bootstrap.cc:654] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.710726 3511 tablet_bootstrap.cc:654] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.711057 3506 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: No bootstrap required, opened a new log
I20260929 03:22:49.711109 3506 ts_tablet_manager.cc:1397] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:49.711097 3129 tablet_service.cc:1467] Processing CreateTablet for tablet 1573ac6ed7ac425fa183415033aa74ff (DEFAULT_TABLE table=test-table [id=308fce32b54d4b028f6cbfd9ecc5ab4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:22:49.711207 3129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1573ac6ed7ac425fa183415033aa74ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:49.711283 3506 raft_consensus.cc:348] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.711359 3506 raft_consensus.cc:374] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.711383 3506 raft_consensus.cc:729] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad2ec4d9f1444779b91587747d507ed, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.711433 3506 consensus_queue.cc:260] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.711527 3506 ts_tablet_manager.cc:1428] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:49.712478 3505 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: Bootstrap starting.
I20260929 03:22:49.712811 3505 tablet_bootstrap.cc:654] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:49.712823 3511 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: No bootstrap required, opened a new log
I20260929 03:22:49.712877 3511 ts_tablet_manager.cc:1397] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:22:49.712988 3511 raft_consensus.cc:348] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.713075 3511 raft_consensus.cc:374] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.713104 3511 raft_consensus.cc:729] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1065acf86647f398a3486877ba078d, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.713146 3511 consensus_queue.cc:260] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.713239 3511 ts_tablet_manager.cc:1428] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.713599 3505 tablet_bootstrap.cc:492] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: No bootstrap required, opened a new log
I20260929 03:22:49.713654 3505 ts_tablet_manager.cc:1397] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:49.713773 3505 raft_consensus.cc:348] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.713861 3505 raft_consensus.cc:374] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:49.713886 3505 raft_consensus.cc:729] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Initialized, Role: FOLLOWER
I20260929 03:22:49.713936 3505 consensus_queue.cc:260] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.714031 3505 ts_tablet_manager.cc:1428] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:49.741045 3562 raft_consensus.cc:482] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:22:49.741166 3562 raft_consensus.cc:504] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.741375 3562 leader_election.cc:288] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:49.741575 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573ac6ed7ac425fa183415033aa74ff" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed" is_pre_election: true
I20260929 03:22:49.741569 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573ac6ed7ac425fa183415033aa74ff" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d" is_pre_election: true
I20260929 03:22:49.741684 3430 raft_consensus.cc:2454] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 0.
I20260929 03:22:49.741689 3288 raft_consensus.cc:2454] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 0.
I20260929 03:22:49.741839 3082 leader_election.cc:302] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad2ec4d9f1444779b91587747d507ed, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:49.741976 3562 raft_consensus.cc:2781] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:22:49.742025 3562 raft_consensus.cc:482] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:22:49.742046 3562 raft_consensus.cc:3037] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.742972 3562 raft_consensus.cc:504] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.743108 3562 leader_election.cc:288] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 election: Requested vote from peers 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637), 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233)
I20260929 03:22:49.743290 3430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573ac6ed7ac425fa183415033aa74ff" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad2ec4d9f1444779b91587747d507ed"
I20260929 03:22:49.743304 3288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1573ac6ed7ac425fa183415033aa74ff" candidate_uuid: "e4463e944fd14b969d383f6092aede1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1065acf86647f398a3486877ba078d"
I20260929 03:22:49.743376 3288 raft_consensus.cc:3037] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.743394 3430 raft_consensus.cc:3037] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:49.744021 3430 raft_consensus.cc:2454] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 1.
I20260929 03:22:49.744186 3288 raft_consensus.cc:2454] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4463e944fd14b969d383f6092aede1a in term 1.
I20260929 03:22:49.744212 3082 leader_election.cc:302] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad2ec4d9f1444779b91587747d507ed, e4463e944fd14b969d383f6092aede1a; no voters:
I20260929 03:22:49.744406 3562 raft_consensus.cc:2781] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:49.744477 3562 raft_consensus.cc:686] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 1 LEADER]: Becoming Leader. State: Replica: e4463e944fd14b969d383f6092aede1a, State: Running, Role: LEADER
I20260929 03:22:49.744520 3562 consensus_queue.cc:237] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } }
I20260929 03:22:49.745179 2990 catalog_manager.cc:5495] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a reported cstate change: term changed from 0 to 1, leader changed from <none> to e4463e944fd14b969d383f6092aede1a (127.27.68.65). New cstate: current_term: 1 leader_uuid: "e4463e944fd14b969d383f6092aede1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4463e944fd14b969d383f6092aede1a" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 03:22:49.752709 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:49.750162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.27.68.65:40763)
0929 03:22:49.750202 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:49.750204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:49.750213 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:49.750258 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:49.750259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:49.750269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:49.750337 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:22:49.750421 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.751162 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.751922 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:49.752263 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:49.752333 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:49.752453 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:22:49.752489 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:22:49.752546 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:49.752604 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:22:49.752626 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:22:49.755064 3430 raft_consensus.cc:1261] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Refusing update from remote peer e4463e944fd14b969d383f6092aede1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.755146 3288 raft_consensus.cc:1261] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Refusing update from remote peer e4463e944fd14b969d383f6092aede1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:22:49.755349 3562 consensus_queue.cc:1035] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad2ec4d9f1444779b91587747d507ed" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.755508 3510 consensus_queue.cc:1035] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1065acf86647f398a3486877ba078d" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:22:49.766835 2992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.766942 2992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:22:49.767653 2992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62dd49fe870d4c3ca3bf5b7b96b624af: Sending DeleteTablet for 3 replicas of tablet 1573ac6ed7ac425fa183415033aa74ff
I20260929 03:22:49.767926 3269 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573ac6ed7ac425fa183415033aa74ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53224
I20260929 03:22:49.767978 3129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573ac6ed7ac425fa183415033aa74ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50054
I20260929 03:22:49.767982 3410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1573ac6ed7ac425fa183415033aa74ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:22:49 UTC) from {username='oryx', principal='oryx/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:57390
I20260929 03:22:49.768026 3532 tablet_replica.cc:330] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: stopping tablet replica
I20260929 03:22:49.768081 3532 raft_consensus.cc:2229] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.768119 3532 raft_consensus.cc:2258] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.768153 3533 tablet_replica.cc:330] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: stopping tablet replica
I20260929 03:22:49.768186 3531 tablet_replica.cc:330] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a: stopping tablet replica
I20260929 03:22:49.768214 3533 raft_consensus.cc:2229] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:22:49.768246 3533 raft_consensus.cc:2258] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.768245 3531 raft_consensus.cc:2229] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:22:49.768307 3531 raft_consensus.cc:2258] T 1573ac6ed7ac425fa183415033aa74ff P e4463e944fd14b969d383f6092aede1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:22:49.768558 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3064
I20260929 03:22:49.768652 3532 ts_tablet_manager.cc:1905] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.769256 3533 ts_tablet_manager.cc:1905] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:22:49.770892 3532 ts_tablet_manager.cc:1918] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:49.771013 3532 log.cc:1196] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/1573ac6ed7ac425fa183415033aa74ff
I20260929 03:22:49.771003 3533 ts_tablet_manager.cc:1918] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:22:49.771055 3533 log.cc:1196] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/1573ac6ed7ac425fa183415033aa74ff
I20260929 03:22:49.771332 3533 ts_tablet_manager.cc:1939] T 1573ac6ed7ac425fa183415033aa74ff P 3ad2ec4d9f1444779b91587747d507ed: Deleting consensus metadata
I20260929 03:22:49.771430 3532 ts_tablet_manager.cc:1939] T 1573ac6ed7ac425fa183415033aa74ff P 0d1065acf86647f398a3486877ba078d: Deleting consensus metadata
I20260929 03:22:49.771682 2978 catalog_manager.cc:4841] TS 3ad2ec4d9f1444779b91587747d507ed (127.27.68.67:44233): tablet 1573ac6ed7ac425fa183415033aa74ff (table test-table [id=308fce32b54d4b028f6cbfd9ecc5ab4b]) successfully deleted
I20260929 03:22:49.771790 2977 catalog_manager.cc:4841] TS 0d1065acf86647f398a3486877ba078d (127.27.68.66:36637): tablet 1573ac6ed7ac425fa183415033aa74ff (table test-table [id=308fce32b54d4b028f6cbfd9ecc5ab4b]) successfully deleted
W20260929 03:22:49.776044 2978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:40763 (error 108)
W20260929 03:22:49.776687 3594 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:49.776475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40763 (local address 127.0.0.1:50078)
0929 03:22:49.776584 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:49.776597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:49.776628 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40763: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
W20260929 03:22:49.776808 2978 catalog_manager.cc:4584] TS e4463e944fd14b969d383f6092aede1a (127.27.68.65:40763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1573ac6ed7ac425fa183415033aa74ff: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40763: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260929 03:22:49.777165 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3204
I20260929 03:22:49.783309 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3344
I20260929 03:22:49.789180 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 2960
2026-09-29T03:22:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2721 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[3597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[3597](info): set up 2 sockets
Sep 29 03:22:49 dist-test-slave-fzwk krb5kdc[3597](info): commencing operation
krb5kdc: starting...
W20260929 03:22:50.828181 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:50 dist-test-slave-fzwk krb5kdc[3597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:50.950346 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33559
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:33559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:51.033082 3610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:51.033319 3610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:51.033344 3610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:51.034555 3610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:51.034602 3610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:51.034616 3610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:51.034654 3610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:51.034682 3610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:51.036259 3610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:33559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:33559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:51.036536 3610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:51.036777 3610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:22:51.041806 3610 server_base.cc:1034] running on GCE node
W20260929 03:22:51.041719 3616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:51.041738 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:51.041752 3618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:51.042171 3610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:51.042495 3610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:51.043649 3610 hybrid_clock.cc:648] HybridClock initialized: now 1790652171043629 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:22:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:22:51 dist-test-slave-fzwk krb5kdc[3623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:22:51 dist-test-slave-fzwk krb5kdc[3623](info): set up 2 sockets
Sep 29 03:22:51 dist-test-slave-fzwk krb5kdc[3623](info): commencing operation
krb5kdc: starting...
W20260929 03:22:52.092156 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:22:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:22:52.225529 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34337
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:52.308863 3636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:52.309123 3636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:52.309162 3636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:52.310329 3636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:52.310405 3636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:52.310428 3636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:52.310448 3636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:52.310479 3636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:52.311995 3636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:52.312301 3636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:52.312537 3636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:52.317402 3644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.317517 3636 server_base.cc:1034] running on GCE node
W20260929 03:22:52.317365 3641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.317370 3642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.317874 3636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:52.318238 3636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:52.319404 3636 hybrid_clock.cc:648] HybridClock initialized: now 1790652172319391 us; error 46 us; skew 500 ppm
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:52.322003 3636 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:52.322520 3636 webserver.cc:460] Webserver started at http://127.27.68.126:45417/ using document root <none> and password file <none>
I20260929 03:22:52.322778 3636 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:52.322849 3636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:52.323010 3636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:52.324189 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "9919ba1a2a8a426daa0c792e72b1ec6d"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "ef75b3fe96acc5ef20f45628a8260955"
server_key_iv: "ac63bd360bc7e36754d6690bc18da0a0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.324572 3636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "9919ba1a2a8a426daa0c792e72b1ec6d"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "ef75b3fe96acc5ef20f45628a8260955"
server_key_iv: "ac63bd360bc7e36754d6690bc18da0a0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.325950 3636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:52.326699 3651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.326884 3636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:52.326987 3636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "9919ba1a2a8a426daa0c792e72b1ec6d"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "ef75b3fe96acc5ef20f45628a8260955"
server_key_iv: "ac63bd360bc7e36754d6690bc18da0a0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.327087 3636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:52.340890 3636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:52.343376 3636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:52.343541 3636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:52.347681 3636 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:34337
I20260929 03:22:52.347718 3703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:34337 every 8 connection(s)
I20260929 03:22:52.348098 3636 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:52.348980 3704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:22:52.350560 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3636
I20260929 03:22:52.350672 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:22:52.350869 27921 external_mini_cluster.cc:1428] Setting key c55f99d4bc86efc50ade7c02820c237f
I20260929 03:22:52.351811 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Bootstrap starting.
I20260929 03:22:52.352521 3704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Neither blocks nor log segments found. Creating new log.
I20260929 03:22:52.352864 3704 log.cc:824] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:52.353615 3704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: No bootstrap required, opened a new log
I20260929 03:22:52.355533 3704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:52.355706 3704 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:22:52.355753 3704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9919ba1a2a8a426daa0c792e72b1ec6d, State: Initialized, Role: FOLLOWER
I20260929 03:22:52.356212 3704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:52.356317 3704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:52.356364 3704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:52.356417 3704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:22:52.356992 3704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:52.357111 3704 leader_election.cc:302] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9919ba1a2a8a426daa0c792e72b1ec6d; no voters:
I20260929 03:22:52.357306 3704 leader_election.cc:288] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:22:52.357369 3709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:22:52.357527 3709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 LEADER]: Becoming Leader. State: Replica: 9919ba1a2a8a426daa0c792e72b1ec6d, State: Running, Role: LEADER
I20260929 03:22:52.357582 3704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:52.357631 3709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:52.357942 3709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9919ba1a2a8a426daa0c792e72b1ec6d. Latest consensus state: current_term: 1 leader_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } } }
I20260929 03:22:52.357955 3711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } } }
I20260929 03:22:52.358050 3709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:52.358071 3711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:52.359045 3718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:52.359588 3718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:52.361303 3718 catalog_manager.cc:1347] Generated new cluster ID: bcfc2a6ef09748deb8a3139994b27e2d
I20260929 03:22:52.361361 3718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:52.362282 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.351462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.27.68.126:34337)
0929 03:22:52.351635 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:52.351648 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:52.351678 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:22:52.352149 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:52.352158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:52.352220 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:52.352345 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:52.353153 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.354172 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.354945 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.355418 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.355548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.358327 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:52.358332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:52.358332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:52.358345 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:52.360484 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.360932 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.360935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.360935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.361028 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.361314 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.361315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.361315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.361640 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:52.361805 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:52.361940 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:22:52.362102 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260929 03:22:52.388882 3718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:22:52.389518 3718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:52.394842 3718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Generated new TSK 0
I20260929 03:22:52.395084 3718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:22:52.426386 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:52.510812 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:52.511018 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:52.511060 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:52.512233 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:52.512274 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:52.512300 3730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:52.513664 3730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:52.513944 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:52.514140 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:52.520879 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.520845 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.521019 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.521173 3730 server_base.cc:1034] running on GCE node
I20260929 03:22:52.521463 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:52.521740 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:52.522897 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790652172522873 us; error 43 us; skew 500 ppm
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:52.525426 3730 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:22:52.525872 3730 webserver.cc:460] Webserver started at http://127.27.68.65:36425/ using document root <none> and password file <none>
I20260929 03:22:52.526065 3730 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:52.526105 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:52.526194 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:52.527201 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "fc273bbfc1b14b25ae70e5e628ca9735"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "fccc42c935e3e4b2d6b0afa6b6101b8e"
server_key_iv: "2c0e3c67830a33aaa13e7d001ee93d6d"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.527601 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "fc273bbfc1b14b25ae70e5e628ca9735"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "fccc42c935e3e4b2d6b0afa6b6101b8e"
server_key_iv: "2c0e3c67830a33aaa13e7d001ee93d6d"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.528859 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:52.529528 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.529716 3730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:22:52.529810 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "fc273bbfc1b14b25ae70e5e628ca9735"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "fccc42c935e3e4b2d6b0afa6b6101b8e"
server_key_iv: "2c0e3c67830a33aaa13e7d001ee93d6d"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.529889 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:52.548656 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:52.551113 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:52.551338 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:52.551602 3730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:52.551939 3730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:52.551972 3730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.551994 3730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:52.552042 3730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.557819 3730 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:32977
I20260929 03:22:52.557875 3858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:32977 every 8 connection(s)
I20260929 03:22:52.558248 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:52.561637 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3730
I20260929 03:22:52.561748 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:22:52.561928 27921 external_mini_cluster.cc:1428] Setting key d6e668e31fc9ce98fc9a858c9c3a31a4
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:52.568673 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.559015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38579 (local address 127.27.68.126:34337)
0929 03:22:52.559067 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:52.559070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:52.559313 (+ 243us) server_negotiation.cc:407] Connection header received
0929 03:22:52.559859 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:52.559860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:52.559870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:52.559931 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:52.560859 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.561356 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.562839 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.563039 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.563098 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.565431 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:52.565434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:52.565434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:52.565443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:52.567143 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.567606 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.567608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.567608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.567681 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.567942 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.567944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.567944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.568113 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:52.568202 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:52.568456 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 03:22:52.568555 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:22:52.568750 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.559006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:38579)
0929 03:22:52.559166 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:52.559208 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:52.559768 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:52.559993 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:52.559996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:52.560575 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:52.560773 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.560777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.561484 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.561486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.562753 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.562757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.563127 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.563128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.563310 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.563844 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:52.563854 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:52.565334 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:52.567291 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.567300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.567301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.567530 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.567760 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.567761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.567761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.567869 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.568186 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:52.568192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:52.568316 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 03:22:52.568405 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:22:52.568551 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 03:22:52.569262 3859 heartbeater.cc:347] Connected to a master server at 127.27.68.126:34337
I20260929 03:22:52.569355 3859 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:52.569550 3859 heartbeater.cc:511] Master 127.27.68.126:34337 requested a full tablet report, sending...
I20260929 03:22:52.570160 3668 ts_manager.cc:194] Registered new tserver with Master: fc273bbfc1b14b25ae70e5e628ca9735 (127.27.68.65:32977)
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:22:52.571157 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:38579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:22:52.589705 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:52.672410 3864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:52.672631 3864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:52.672653 3864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:52.673873 3864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:52.673921 3864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:52.673949 3864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:52.675465 3864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:52.675740 3864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:52.675988 3864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:52.681137 3872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.681140 3869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.681142 3870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.681435 3864 server_base.cc:1034] running on GCE node
I20260929 03:22:52.681679 3864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:52.681941 3864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:52.683105 3864 hybrid_clock.cc:648] HybridClock initialized: now 1790652172683080 us; error 35 us; skew 500 ppm
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:52.685683 3864 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:22:52.686208 3864 webserver.cc:460] Webserver started at http://127.27.68.66:43915/ using document root <none> and password file <none>
I20260929 03:22:52.686452 3864 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:52.686527 3864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:52.686679 3864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:52.687851 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "16224953b9dc4afb9c600fd7c082933e"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "5582f3107bcb38eb5215257bde00cf69"
server_key_iv: "0939004effb81d84564bce927a115fa0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.688227 3864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "16224953b9dc4afb9c600fd7c082933e"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "5582f3107bcb38eb5215257bde00cf69"
server_key_iv: "0939004effb81d84564bce927a115fa0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.689587 3864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:52.690503 3879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.690727 3864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:52.690840 3864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "16224953b9dc4afb9c600fd7c082933e"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "5582f3107bcb38eb5215257bde00cf69"
server_key_iv: "0939004effb81d84564bce927a115fa0"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.690917 3864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:52.707785 3864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:52.709865 3864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:52.710038 3864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:52.710282 3864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:52.710616 3864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:52.710675 3864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.710743 3864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:52.710776 3864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.716977 3864 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:40413
I20260929 03:22:52.717020 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:40413 every 8 connection(s)
I20260929 03:22:52.717388 3864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:52.725005 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3864
I20260929 03:22:52.725140 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:22:52.725373 27921 external_mini_cluster.cc:1428] Setting key 7fa8d93a51e112c1783f0f51f42ae543
I20260929 03:22:52.727078 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.718181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:41385 (local address 127.27.68.126:34337)
0929 03:22:52.718245 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:52.718247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:52.718471 (+ 224us) server_negotiation.cc:407] Connection header received
0929 03:22:52.718952 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:52.718953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:52.718962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:52.719053 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:52.720106 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.720620 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.721638 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.721853 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.721943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.724146 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:52.724148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:52.724148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:52.724156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:52.725650 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.726076 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.726078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.726078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.726144 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.726410 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.726411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.726411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.726534 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:52.726612 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:52.726903 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 03:22:52.726985 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:22:52.727322 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.718159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:41385)
0929 03:22:52.718336 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:52.718379 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:52.718868 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:52.719146 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:52.719149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:52.719801 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:52.719999 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.720004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.720726 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.720727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.721539 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.721541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.721954 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.721955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.722085 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.722621 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:52.722630 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:52.724035 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:52.725751 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.725759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.725761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.725998 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.726216 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.726216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.726217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.726335 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.726602 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:52.726608 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:52.726766 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 03:22:52.726874 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:22:52.727040 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260929 03:22:52.727813 3993 heartbeater.cc:347] Connected to a master server at 127.27.68.126:34337
I20260929 03:22:52.727908 3993 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:52.728111 3993 heartbeater.cc:511] Master 127.27.68.126:34337 requested a full tablet report, sending...
I20260929 03:22:52.728518 3668 ts_manager.cc:194] Registered new tserver with Master: 16224953b9dc4afb9c600fd7c082933e (127.27.68.66:40413)
I20260929 03:22:52.728856 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:41385
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:22:52.750331 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:52.832604 3998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:52.832860 3998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:52.832904 3998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:52.834177 3998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:52.834262 3998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:52.834313 3998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:52.835952 3998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:52.836246 3998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:52.836495 3998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:52.841665 4003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.841777 3998 server_base.cc:1034] running on GCE node
W20260929 03:22:52.841645 4004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:52.841642 4006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:52.842123 3998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:52.842401 3998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:52.843573 3998 hybrid_clock.cc:648] HybridClock initialized: now 1790652172843558 us; error 31 us; skew 500 ppm
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:52.846439 3998 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:22:52.846931 3998 webserver.cc:460] Webserver started at http://127.27.68.67:44073/ using document root <none> and password file <none>
I20260929 03:22:52.847183 3998 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:52.847293 3998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:52.847464 3998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:22:52.848579 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "5c11ca2386164f61a67221d77f9797ca"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "af003d4a4783ea7bb25f9d01c0777c97"
server_key_iv: "b378ee589cad729ed985551a4b9b2d08"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.848963 3998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "5c11ca2386164f61a67221d77f9797ca"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "af003d4a4783ea7bb25f9d01c0777c97"
server_key_iv: "b378ee589cad729ed985551a4b9b2d08"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.850400 3998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:22:52.851192 4013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.851444 3998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:52.851547 3998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "5c11ca2386164f61a67221d77f9797ca"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "af003d4a4783ea7bb25f9d01c0777c97"
server_key_iv: "b378ee589cad729ed985551a4b9b2d08"
server_key_version: "encryptionkey@0"
I20260929 03:22:52.851632 3998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:52.860167 3998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:52.862183 3998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:52.862362 3998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:52.862601 3998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:52.862934 3998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:52.862989 3998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.863032 3998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:52.863058 3998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:52.869136 3998 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:34325
I20260929 03:22:52.869176 4126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:34325 every 8 connection(s)
I20260929 03:22:52.869556 3998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:22:52.875275 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 3998
I20260929 03:22:52.875396 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:22:52.875640 27921 external_mini_cluster.cc:1428] Setting key 852a17606da9c0519875b72bea5d56bd
Sep 29 03:22:52 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:22:52.879705 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.870327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:54925 (local address 127.27.68.126:34337)
0929 03:22:52.870395 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:52.870397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:52.870627 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:22:52.871153 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:52.871154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:52.871165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:52.871300 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:52.872339 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.872836 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.873879 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.874062 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.874113 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.876687 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:52.876688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:52.876689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:52.876705 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:52.878133 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.878593 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.878600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.878600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.878655 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:52.878935 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:52.878936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:52.878936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:52.879059 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:52.879132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:52.879508 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:22:52.879599 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:22:52.879913 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:52.870316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:54925)
0929 03:22:52.870511 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:52.870552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:52.871053 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:52.871370 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:52.871374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:52.872012 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:52.872242 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.872248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.872937 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.872938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.873789 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:52.873792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:52.874142 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:52.874143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:52.874252 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:52.874793 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:52.874801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:52.876565 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:22:52.878234 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.878244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.878247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.878498 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.878728 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:22:52.878729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:22:52.878729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:22:52.878857 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:22:52.879133 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:22:52.879140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:22:52.879367 (+ 227us) client_negotiation.cc:769] Sending connection context
0929 03:22:52.879483 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:22:52.879656 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260929 03:22:52.880452 4127 heartbeater.cc:347] Connected to a master server at 127.27.68.126:34337
I20260929 03:22:52.880563 4127 heartbeater.cc:464] Registering TS with master...
I20260929 03:22:52.880719 4127 heartbeater.cc:511] Master 127.27.68.126:34337 requested a full tablet report, sending...
I20260929 03:22:52.881124 3668 ts_manager.cc:194] Registered new tserver with Master: 5c11ca2386164f61a67221d77f9797ca (127.27.68.67:34325)
I20260929 03:22:52.881471 3668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:54925
I20260929 03:22:52.887617 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
Entry for principal oryx/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab.
WARNING: no policy specified for oryx/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
Entry for principal oryx/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab.
WARNING: no policy specified for oryx/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
Entry for principal oryx/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab.
I20260929 03:22:52.961642 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3730
I20260929 03:22:52.966244 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3864
I20260929 03:22:52.970795 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3998
I20260929 03:22:52.975049 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 3636
I20260929 03:22:52.980473 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34337
--webserver_interface=127.27.68.126
--webserver_port=45417
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:22:53.065716 4136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:53.065992 4136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:53.066035 4136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:53.067268 4136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:22:53.067343 4136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:53.067368 4136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:53.067399 4136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:22:53.067440 4136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:22:53.069062 4136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:53.069357 4136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:53.069633 4136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:53.074672 4141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.074699 4142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.074721 4144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.074877 4136 server_base.cc:1034] running on GCE node
I20260929 03:22:53.075192 4136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:53.075538 4136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:53.076712 4136 hybrid_clock.cc:648] HybridClock initialized: now 1790652173076705 us; error 35 us; skew 500 ppm
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:53.079367 4136 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:22:53.079820 4136 webserver.cc:460] Webserver started at http://127.27.68.126:45417/ using document root <none> and password file <none>
I20260929 03:22:53.080075 4136 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:53.080168 4136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:53.081529 4136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:53.082181 4151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.082350 4136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:53.082459 4136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "9919ba1a2a8a426daa0c792e72b1ec6d"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "ef75b3fe96acc5ef20f45628a8260955"
server_key_iv: "ac63bd360bc7e36754d6690bc18da0a0"
server_key_version: "encryptionkey@0"
I20260929 03:22:53.082803 4136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:53.096850 4136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:53.098889 4136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:53.099113 4136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:53.102872 4136 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:34337
I20260929 03:22:53.102948 4203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:34337 every 8 connection(s)
I20260929 03:22:53.103398 4136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:22:53.104878 4204 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:22:53.105571 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Bootstrap starting.
I20260929 03:22:53.107669 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4136
I20260929 03:22:53.107976 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:32977
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:34337
--webserver_port=36425
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:22:53.114555 4204 log.cc:824] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:22:53.116472 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:22:53.116809 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Bootstrap complete.
I20260929 03:22:53.119405 4204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:53.119686 4204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9919ba1a2a8a426daa0c792e72b1ec6d, State: Initialized, Role: FOLLOWER
I20260929 03:22:53.119892 4204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:53.119976 4204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:22:53.120030 4204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:22:53.120085 4204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:22:53.121184 4204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:53.121323 4204 leader_election.cc:302] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9919ba1a2a8a426daa0c792e72b1ec6d; no voters:
I20260929 03:22:53.121542 4204 leader_election.cc:288] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:22:53.121582 4208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:22:53.121699 4208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [term 2 LEADER]: Becoming Leader. State: Replica: 9919ba1a2a8a426daa0c792e72b1ec6d, State: Running, Role: LEADER
I20260929 03:22:53.121785 4208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } }
I20260929 03:22:53.121848 4204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:22:53.122076 4208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9919ba1a2a8a426daa0c792e72b1ec6d. Latest consensus state: current_term: 2 leader_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } } }
I20260929 03:22:53.122152 4208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:53.122081 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9919ba1a2a8a426daa0c792e72b1ec6d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34337 } } }
I20260929 03:22:53.122431 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d [sys.catalog]: This master's current role is: LEADER
I20260929 03:22:53.122642 4212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:22:53.123204 4212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:22:53.123616 4212 catalog_manager.cc:1259] Loaded cluster ID: bcfc2a6ef09748deb8a3139994b27e2d
I20260929 03:22:53.123667 4212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:22:53.124940 4212 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:22:53.125133 4212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9919ba1a2a8a426daa0c792e72b1ec6d: Loaded TSK: 0
I20260929 03:22:53.125370 4212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:22:53.194479 4206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:53.194716 4206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:53.194742 4206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:53.196135 4206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:53.196188 4206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:53.196218 4206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:22:53.197768 4206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.65.keytab
--principal=oryx/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:32977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=36425
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:53.198037 4206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:53.198297 4206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:53.203521 4233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.203622 4206 server_base.cc:1034] running on GCE node
W20260929 03:22:53.203516 4230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.203749 4231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.203972 4206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:53.204262 4206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:53.205423 4206 hybrid_clock.cc:648] HybridClock initialized: now 1790652173205411 us; error 28 us; skew 500 ppm
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652173, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:53.208066 4206 init.cc:377] Logged in from keytab as oryx/127.27.68.65@KRBTEST.COM (short username oryx)
I20260929 03:22:53.208612 4206 webserver.cc:460] Webserver started at http://127.27.68.65:36425/ using document root <none> and password file <none>
I20260929 03:22:53.208868 4206 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:53.208926 4206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:53.210362 4206 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:22:53.211012 4240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.211162 4206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:22:53.211279 4206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "fc273bbfc1b14b25ae70e5e628ca9735"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "fccc42c935e3e4b2d6b0afa6b6101b8e"
server_key_iv: "2c0e3c67830a33aaa13e7d001ee93d6d"
server_key_version: "encryptionkey@0"
I20260929 03:22:53.211642 4206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:53.229699 4206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:53.232082 4206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:53.232254 4206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:53.232544 4206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:53.232913 4206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:53.232947 4206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.232970 4206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:53.233003 4206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.240475 4206 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:32977
I20260929 03:22:53.240634 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:32977 every 8 connection(s)
I20260929 03:22:53.240967 4206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:22:53.243680 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4206
I20260929 03:22:53.244007 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:40413
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:34337
--webserver_port=43915
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.252230 4356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.241710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:49761)
0929 03:22:53.241939 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.241991 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.242640 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.243332 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.243336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.243994 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.244338 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.244345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.245708 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.245709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.246979 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.246982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.248017 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.248019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.248237 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.248945 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.248955 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.251987 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
W20260929 03:22:53.252501 4354 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:34337 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.68.126:34337: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
W20260929 03:22:53.252820 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.241828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:49761 (local address 127.27.68.126:34337)
0929 03:22:53.242860 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.242869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.242904 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:22:53.243027 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.243032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.243085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.243231 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.244472 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.245571 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.247092 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.247841 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.248021 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.252191 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:49761: BlockingRecv error: recv got EOF from 127.27.68.65:49761 (error 108)
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":106,"threads_started":1}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.258185 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.252754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:52383)
0929 03:22:53.252943 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.252954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.253016 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.253361 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.253362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.253505 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.253556 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.253558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.254563 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.254564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.255092 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.255093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.255617 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.255617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.255668 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.256167 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.256171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.258092 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":125,"threads_started":1}
W20260929 03:22:53.258277 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.252794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52383 (local address 127.27.68.126:34337)
0929 03:22:53.253217 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.253220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.253230 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:53.253258 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.253259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.253275 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.253354 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.253636 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.254465 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.255172 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.255524 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.255598 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.258190 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:52383: BlockingRecv error: recv got EOF from 127.27.68.65:52383 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":78,"threads_started":1}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.263528 4359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.258667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:46523)
0929 03:22:53.258779 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.258791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.258847 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.258966 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.258967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.259120 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.259163 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.259165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.259812 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.259813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.260659 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.260661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.260991 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.260992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.261058 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.261478 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.261486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.263441 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 03:22:53.263597 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.258733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46523 (local address 127.27.68.126:34337)
0929 03:22:53.258787 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.258789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.258830 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:22:53.258906 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.258906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.258915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.258959 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.259274 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.259733 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.260743 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.260921 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.260974 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.263528 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:46523: BlockingRecv error: recv got EOF from 127.27.68.65:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 03:22:53.263833 4354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:22:53.332063 4358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:53.332299 4358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:53.332325 4358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:53.333554 4358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:53.333603 4358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:53.333631 4358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:22:53.335153 4358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.66.keytab
--principal=oryx/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:40413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=43915
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:53.335407 4358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:53.335608 4358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:53.340684 4365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.340714 4368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.340715 4358 server_base.cc:1034] running on GCE node
W20260929 03:22:53.340684 4366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.341120 4358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:53.341395 4358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:53.342602 4358 hybrid_clock.cc:648] HybridClock initialized: now 1790652173342596 us; error 54 us; skew 500 ppm
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652173, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:53.349695 4358 init.cc:377] Logged in from keytab as oryx/127.27.68.66@KRBTEST.COM (short username oryx)
I20260929 03:22:53.350212 4358 webserver.cc:460] Webserver started at http://127.27.68.66:43915/ using document root <none> and password file <none>
I20260929 03:22:53.350474 4358 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:53.350565 4358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:53.352118 4358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:53.352787 4375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.353016 4358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:22:53.353116 4358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "16224953b9dc4afb9c600fd7c082933e"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "5582f3107bcb38eb5215257bde00cf69"
server_key_iv: "0939004effb81d84564bce927a115fa0"
server_key_version: "encryptionkey@0"
I20260929 03:22:53.353550 4358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:53.367182 4358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:53.369864 4358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:53.370023 4358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:53.370241 4358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:53.370507 4358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:53.370541 4358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.370566 4358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:53.370582 4358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.377170 4358 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:40413
I20260929 03:22:53.377213 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:40413 every 8 connection(s)
I20260929 03:22:53.377622 4358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:22:53.379568 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4358
I20260929 03:22:53.379948 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:34325
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:34337
--webserver_port=44073
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:22:53.386297 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.378490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51087 (local address 127.27.68.126:34337)
0929 03:22:53.378550 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.378553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.378764 (+ 211us) server_negotiation.cc:407] Connection header received
0929 03:22:53.379333 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.379333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.379348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.379406 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.380869 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.381371 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.382457 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.382666 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.382757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.386238 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:51087: BlockingRecv error: recv got EOF from 127.27.68.66:51087 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:22:53.386369 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.378445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:51087)
0929 03:22:53.378628 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.378673 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.379170 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.379494 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.379499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.380398 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.380714 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.380722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.381514 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.381515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.382351 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.382353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.382768 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.382770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.382904 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.383541 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.383551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.386150 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260929 03:22:53.386598 4489 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:34337 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.68.126:34337: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.391511 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.386846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:40343)
0929 03:22:53.386881 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.386891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.386933 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.387117 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.387118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.387332 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.387379 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.387381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.388056 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.388058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.388586 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.388588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.388960 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.388961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.389029 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.389498 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.389504 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.391432 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 03:22:53.391624 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.386911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:40343 (local address 127.27.68.126:34337)
0929 03:22:53.386979 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.386981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.386990 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:53.387017 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.387017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.387027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.387100 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.387467 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.387959 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.388674 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.388878 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.388960 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.391527 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:40343: BlockingRecv error: recv got EOF from 127.27.68.66:40343 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.396255 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.391870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:34947)
0929 03:22:53.391923 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.391933 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.391996 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.392133 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.392134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.392272 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.392301 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.392303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.392942 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.392943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.393439 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.393440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.393808 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.393809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.393861 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.394322 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.394326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.396178 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:22:53.396421 4489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:22:53.396560 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.391941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:34947 (local address 127.27.68.126:34337)
0929 03:22:53.391999 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.392001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.392010 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:53.392055 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.392056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.392074 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.392129 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.392378 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.392858 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.393519 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.393725 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.393807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.396274 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:34947: BlockingRecv error: recv got EOF from 127.27.68.66:34947 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 03:22:53.465751 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:22:53.466008 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:22:53.466053 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:22:53.467348 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:22:53.467410 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:22:53.467453 4492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:22:53.468998 4492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/krb5kdc/oryx_127.27.68.67.keytab
--principal=oryx/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=44073
--tserver_master_addrs=127.27.68.126:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:22:53.469278 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:22:53.469547 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:22:53.474809 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.474807 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:22:53.474807 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:22:53.475471 4492 server_base.cc:1034] running on GCE node
I20260929 03:22:53.475719 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:22:53.475996 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:22:53.477173 4492 hybrid_clock.cc:648] HybridClock initialized: now 1790652173477152 us; error 41 us; skew 500 ppm
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652173, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:22:53.479836 4492 init.cc:377] Logged in from keytab as oryx/127.27.68.67@KRBTEST.COM (short username oryx)
I20260929 03:22:53.480288 4492 webserver.cc:460] Webserver started at http://127.27.68.67:44073/ using document root <none> and password file <none>
I20260929 03:22:53.480551 4492 fs_manager.cc:359] Metadata directory not provided
I20260929 03:22:53.480756 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:22:53.482123 4492 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:22:53.482657 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.482815 4492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.482915 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "5c11ca2386164f61a67221d77f9797ca"
format_stamp: "Formatted at 2026-09-29 03:22:52 on dist-test-slave-fzwk"
server_key: "af003d4a4783ea7bb25f9d01c0777c97"
server_key_iv: "b378ee589cad729ed985551a4b9b2d08"
server_key_version: "encryptionkey@0"
I20260929 03:22:53.483321 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:22:53.506007 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:22:53.508488 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:22:53.508664 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:22:53.508903 4492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:22:53.509222 4492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:22:53.509272 4492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.509306 4492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:22:53.509332 4492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:22:53.515554 4492 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:34325
I20260929 03:22:53.515605 4620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:34325 every 8 connection(s)
I20260929 03:22:53.515969 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:22:53.523622 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.516739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:43859 (local address 127.27.68.126:34337)
0929 03:22:53.516808 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.516810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.517037 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:22:53.517551 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.517552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.517562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.517620 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.518610 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.519099 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.520109 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.520303 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.520359 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.523566 (+ 3207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:43859: BlockingRecv error: recv got EOF from 127.27.68.67:43859 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:22:53.523690 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.516724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:43859)
0929 03:22:53.516903 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.516947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.517436 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.517689 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.517693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.518300 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.518517 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.518523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.519184 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.519184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.520025 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.520027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.520380 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.520381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.520486 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.520961 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.520970 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.523467 (+ 2497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
W20260929 03:22:53.523939 4621 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:34337 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.68.126:34337: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
I20260929 03:22:53.525563 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4492
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.528992 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.524205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:45665)
0929 03:22:53.524259 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.524275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.524309 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.524548 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.524549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.524682 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.524729 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.524731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.525413 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.525414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.525970 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.525973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.526495 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.526496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.526548 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.527015 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.527020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.528899 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:22:53.529088 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.524371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45665 (local address 127.27.68.126:34337)
0929 03:22:53.524425 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.524427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.524433 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:22:53.524455 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.524455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.524464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.524543 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.524813 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.525302 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.526094 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.526424 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.526499 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.529019 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:45665: BlockingRecv error: recv got EOF from 127.27.68.67:45665 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:53 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:53.533881 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.525999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.27.68.126:34337)
0929 03:22:53.526060 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.526062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.526244 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:22:53.526300 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.526301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.526310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.526367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.527016 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.527577 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.528578 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.528881 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.528952 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.530271 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:22:53.530281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:22:53.530281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:22:53.530295 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:22:53.532355 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:53.532723 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:53.532728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:53.532729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:53.532867 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:22:53.533097 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:22:53.533098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:22:53.533098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:22:53.533376 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:22:53.533527 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 03:22:53.533660 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:22:53.533778 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:22:53.534336 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:53.529585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:58133)
0929 03:22:53.529656 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:53.529667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:53.529709 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:53.529850 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:53.529851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:53.529988 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:53.530026 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.530027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.530742 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.530743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.531598 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:53.531602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.531935 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.531936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:53.531989 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.532328 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:53.532333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:53.534273 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 03:22:53.534360 4357 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:53.529671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:58133 (local address 127.27.68.126:34337)
0929 03:22:53.529732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:53.529734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:53.529745 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:53.529765 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:53.529766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:53.529775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:53.529846 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:53.530109 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.530658 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.531678 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:53.531861 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:53.531916 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:53.534326 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:58133: BlockingRecv error: recv got EOF from 127.27.68.67:58133 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260929 03:22:53.534564 4621 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:54.268975 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:54.264327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:46287)
0929 03:22:54.264484 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:54.264496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:54.264550 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:54.264680 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:54.264681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:54.264852 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:54.264928 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.264930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.265559 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.265560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.266107 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.266109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.266438 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.266439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.266488 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.266991 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:54.266996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:54.268873 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
W20260929 03:22:54.269045 4626 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:54.264416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46287 (local address 127.27.68.126:34337)
0929 03:22:54.264548 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:54.264550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:54.264559 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:54.264599 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:54.264600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:54.264609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:54.264665 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:54.265001 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.265495 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.266182 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.266373 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.266424 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.268972 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:46287: BlockingRecv error: recv got EOF from 127.27.68.65:46287 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:54.401932 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:54.397019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:34533)
0929 03:22:54.397246 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:54.397259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:54.397357 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:54.397526 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:54.397527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:54.397715 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:54.397761 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.397763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.398449 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.398450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.398999 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.399001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.399469 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.399471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.399536 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.399998 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:54.400003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:54.401828 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":134,"threads_started":1}
W20260929 03:22:54.402117 4626 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:54.397153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:34533 (local address 127.27.68.126:34337)
0929 03:22:54.397251 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:54.397254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:54.397337 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:22:54.397444 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:54.397445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:54.397458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:54.397512 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:54.397842 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.398358 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.399089 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.399381 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.399468 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.401930 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:34533: BlockingRecv error: recv got EOF from 127.27.68.66:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:54 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:54.539656 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:54.534905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:47065)
0929 03:22:54.535060 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:54.535071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:54.535152 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:54.535328 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:54.535329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:54.535509 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:54.535555 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.535556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.536239 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.536240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.536786 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:54.536788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.537184 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.537185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:54.537240 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.537726 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:54.537732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:54.539588 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
W20260929 03:22:54.539733 4626 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:54.534973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:47065 (local address 127.27.68.126:34337)
0929 03:22:54.535042 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:54.535044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:54.535131 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:22:54.535235 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:54.535235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:54.535249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:54.535322 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:54.535653 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.536154 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.536890 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:54.537096 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:54.537182 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:54.539685 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:47065: BlockingRecv error: recv got EOF from 127.27.68.67:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:55.274420 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:55.269570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:46429)
0929 03:22:55.269807 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:55.269819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:55.269881 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:55.270017 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:55.270017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:55.270182 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:55.270237 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.270238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.270897 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.270898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.271459 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.271461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.271807 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.271808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.271846 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.272403 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:55.272408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:55.274360 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":169,"threads_started":1}
W20260929 03:22:55.274578 4630 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:55.269756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46429 (local address 127.27.68.126:34337)
0929 03:22:55.269893 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:55.269895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:55.269904 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:55.269928 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:55.269928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:55.269939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:55.270000 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:55.270305 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.270825 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.271546 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.271738 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.271793 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.274477 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:46429: BlockingRecv error: recv got EOF from 127.27.68.65:46429 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:55.407581 4631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:55.402629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:39217)
0929 03:22:55.402795 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:55.402808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:55.402912 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:55.403064 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:55.403065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:55.403308 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:55.403356 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.403358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.404049 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.404050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.404817 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.404820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.405213 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.405214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.405273 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.405721 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:55.405726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:55.407488 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
W20260929 03:22:55.407653 4630 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:55.402731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:39217 (local address 127.27.68.126:34337)
0929 03:22:55.402786 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:55.402789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:55.402896 (+ 107us) server_negotiation.cc:407] Connection header received
0929 03:22:55.402985 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:55.402986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:55.403001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:55.403048 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:55.403436 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.403956 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.404925 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.405130 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.405208 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.407587 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:39217: BlockingRecv error: recv got EOF from 127.27.68.66:39217 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:55 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:55.545266 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:55.540182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:35339)
0929 03:22:55.540320 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:55.540331 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:55.540415 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:55.540574 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:55.540575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:55.540749 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:55.540799 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.540801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.541601 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.541602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.542173 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:55.542176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.542571 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.542572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:55.542635 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.543118 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:55.543123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:55.545206 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260929 03:22:55.545389 4630 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:55.540349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:35339 (local address 127.27.68.126:34337)
0929 03:22:55.540414 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:55.540416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:55.540427 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:22:55.540483 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:55.540484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:55.540501 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:55.540565 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:55.540885 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.541505 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.542300 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:55.542497 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:55.542573 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:55.545310 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:35339: BlockingRecv error: recv got EOF from 127.27.68.67:35339 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:56.279844 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:56.274933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:52123)
0929 03:22:56.275060 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:56.275070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:56.275120 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:56.275324 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:56.275324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:56.275483 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:56.275535 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.275537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.276218 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.276219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.276745 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.276746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.277085 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.277086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.277128 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.277738 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:56.277742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:56.279782 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260929 03:22:56.280014 4634 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:56.275019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52123 (local address 127.27.68.126:34337)
0929 03:22:56.275152 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:56.275154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:56.275164 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:22:56.275189 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:56.275190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:56.275200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:56.275290 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:56.275616 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.276141 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.276821 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.277020 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.277071 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.279920 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:52123: BlockingRecv error: recv got EOF from 127.27.68.65:52123 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:56.412690 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:56.408097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:41513)
0929 03:22:56.408235 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:56.408245 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:56.408303 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:56.408441 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:56.408442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:56.408610 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:56.408659 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.408660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.409291 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.409292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.409810 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.409811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.410314 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.410315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.410388 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.410857 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:56.410862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:56.412624 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260929 03:22:56.412819 4634 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:56.408179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:41513 (local address 127.27.68.126:34337)
0929 03:22:56.408257 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:56.408259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:56.408289 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:22:56.408351 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:56.408352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:56.408362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:56.408421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:56.408727 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.409219 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.409912 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.410219 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.410297 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.412737 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:41513: BlockingRecv error: recv got EOF from 127.27.68.66:41513 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:56 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:56.550493 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:56.545702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:49101)
0929 03:22:56.545895 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:56.545907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:56.545998 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:56.546158 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:56.546158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:56.546336 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:56.546382 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.546384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.547050 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.547051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.547661 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:56.547663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.548038 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.548039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:56.548096 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.548596 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:56.548602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:56.550386 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
W20260929 03:22:56.550529 4634 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:56.545796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:49101 (local address 127.27.68.126:34337)
0929 03:22:56.545886 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:56.545888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:56.545978 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:22:56.546063 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:56.546064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:56.546078 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:56.546148 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:56.546464 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.546962 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.547768 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:56.547955 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:56.548037 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:56.550463 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:49101: BlockingRecv error: recv got EOF from 127.27.68.67:49101 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:57.285136 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:57.280330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:34203)
0929 03:22:57.280488 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:57.280501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:57.280573 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:57.280760 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:57.280761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:57.280947 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:57.280995 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.280997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.281684 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.281686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.282241 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.282243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.282625 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.282627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.282688 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.283166 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:57.283170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:57.285076 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
W20260929 03:22:57.285279 4638 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:57.280375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34203 (local address 127.27.68.126:34337)
0929 03:22:57.280507 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:57.280509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:57.280562 (+ 53us) server_negotiation.cc:407] Connection header received
0929 03:22:57.280658 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:57.280659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:57.280671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:57.280756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:57.281080 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.281596 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.282334 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.282535 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.282623 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.285191 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:34203: BlockingRecv error: recv got EOF from 127.27.68.65:34203 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:57.417914 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:57.413171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:40231)
0929 03:22:57.413319 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:57.413337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:57.413396 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:57.413523 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:57.413523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:57.413686 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:57.413742 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.413744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.414480 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.414481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.415102 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.415103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.415576 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.415577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.415619 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.416087 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:57.416092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:57.417854 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260929 03:22:57.418038 4638 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:57.413247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:40231 (local address 127.27.68.126:34337)
0929 03:22:57.413302 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:57.413304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:57.413380 (+ 76us) server_negotiation.cc:407] Connection header received
0929 03:22:57.413439 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:57.413440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:57.413451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:57.413500 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:57.413836 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.414404 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.415183 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.415469 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.415561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.417954 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:40231: BlockingRecv error: recv got EOF from 127.27.68.66:40231 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:57 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:57.556604 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:57.551193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:57911)
0929 03:22:57.551401 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:57.551412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:57.551515 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:57.551676 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:57.551676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:57.551868 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:57.551917 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.551919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.552648 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.552649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.553179 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:57.553180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.553573 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.553574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:57.553638 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.554179 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:57.554183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:57.556544 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":135,"threads_started":1}
W20260929 03:22:57.556769 4638 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:57.551313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:57911 (local address 127.27.68.126:34337)
0929 03:22:57.551378 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:57.551380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:57.551493 (+ 113us) server_negotiation.cc:407] Connection header received
0929 03:22:57.551582 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:57.551583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:57.551598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:57.551670 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:57.551997 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.552545 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.553281 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:57.553470 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:57.553558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:57.556683 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:57911: BlockingRecv error: recv got EOF from 127.27.68.67:57911 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:58.290496 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:58.285705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:35687)
0929 03:22:58.285838 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:58.285848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:58.285895 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:58.286058 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:58.286059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:58.286275 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:58.286338 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.286340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.287107 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.287107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.287789 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.287791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.288138 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.288139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.288179 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.288645 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:58.288649 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:58.290435 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
W20260929 03:22:58.290621 4642 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:58.285745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:35687 (local address 127.27.68.126:34337)
0929 03:22:58.285913 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:58.285915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:58.285932 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:22:58.285959 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:58.285960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:58.285973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:58.286043 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:58.286440 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.287028 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.287862 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.288068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.288125 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.290524 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:35687: BlockingRecv error: recv got EOF from 127.27.68.65:35687 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:58.429441 4643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:58.418591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:40039)
0929 03:22:58.418760 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:58.418775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:58.418845 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:58.418999 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:58.419000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:58.419171 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:58.419293 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.419295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.419953 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.419953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.420506 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.420508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.420828 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.420829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.420868 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.427421 (+ 6553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:58.427427 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:58.429378 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
W20260929 03:22:58.429600 4642 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:58.418745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:40039 (local address 127.27.68.126:34337)
0929 03:22:58.418817 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:58.418819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:58.418828 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:22:58.418911 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:58.418912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:58.418924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:58.418972 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:58.419372 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.419876 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.420581 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.420764 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.420813 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.429514 (+ 8701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:40039: BlockingRecv error: recv got EOF from 127.27.68.66:40039 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:58 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:58.561764 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:58.557140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:34497)
0929 03:22:58.557285 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:58.557296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:58.557368 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:58.557535 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:58.557536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:58.557711 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:58.557763 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.557765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.558413 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.558414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.558947 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:58.558949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.559355 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.559356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:58.559398 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.559838 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:58.559843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:58.561695 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260929 03:22:58.561933 4642 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:58.557184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:34497 (local address 127.27.68.126:34337)
0929 03:22:58.557249 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:58.557251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:58.557372 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:22:58.557436 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:58.557437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:58.557455 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:58.557516 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:58.557827 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.558316 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.559020 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:58.559276 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:58.559351 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:58.561836 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:34497: BlockingRecv error: recv got EOF from 127.27.68.67:34497 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:59.296096 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:59.291195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:41043)
0929 03:22:59.291396 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:59.291411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:59.291501 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:59.291660 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:59.291660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:59.291820 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:59.291901 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.291903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.292631 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.292633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.293168 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.293170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.293564 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.293565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.293629 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.294099 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:59.294104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:59.296034 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":122,"threads_started":1}
W20260929 03:22:59.296274 4646 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:59.291340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:41043 (local address 127.27.68.126:34337)
0929 03:22:59.291479 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:59.291481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:59.291493 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:22:59.291565 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:59.291566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:59.291577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:59.291672 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:59.291995 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.292537 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.293268 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.293471 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.293561 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.296172 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:41043: BlockingRecv error: recv got EOF from 127.27.68.65:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:59.435047 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:59.429991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:56285)
0929 03:22:59.430179 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:59.430195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:59.430304 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:59.430475 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:59.430476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:59.430660 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:59.430715 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.430717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.431480 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.431481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.432027 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.432029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.432439 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.432440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.432508 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.433021 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:59.433026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:59.434948 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
W20260929 03:22:59.435141 4646 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:59.430202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:56285 (local address 127.27.68.126:34337)
0929 03:22:59.430285 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:59.430287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:59.430299 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:22:59.430373 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:59.430374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:59.430388 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:59.430471 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:59.430804 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.431394 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.432127 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.432347 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.432437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.435069 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:56285: BlockingRecv error: recv got EOF from 127.27.68.66:56285 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:22:59 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:22:59.567145 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:22:59.562387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:40493)
0929 03:22:59.562551 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:22:59.562569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:22:59.562641 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:22:59.562783 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:22:59.562784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:22:59.562975 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:22:59.563029 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.563031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.563717 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.563718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.564282 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:22:59.564285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.564615 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.564616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:22:59.564656 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.565115 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:22:59.565120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:22:59.567087 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260929 03:22:59.567356 4646 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:22:59.562499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:40493 (local address 127.27.68.126:34337)
0929 03:22:59.562551 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:22:59.562553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:22:59.562643 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:22:59.562699 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:22:59.562700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:22:59.562713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:22:59.562763 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:22:59.563096 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.563644 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.564363 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:22:59.564548 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:22:59.564599 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:22:59.567268 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:40493: BlockingRecv error: recv got EOF from 127.27.68.67:40493 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:00.301427 4649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:00.296656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:60399)
0929 03:23:00.296786 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:00.296804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:00.296854 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:00.297024 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:00.297025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:00.297191 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:00.297243 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.297244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.297893 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.297893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.298430 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.298432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.298770 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.298771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.298811 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.299318 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:00.299323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:00.301322 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260929 03:23:00.301512 4650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:00.296731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:60399 (local address 127.27.68.126:34337)
0929 03:23:00.296889 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:00.296890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:00.296900 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:23:00.296924 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:00.296925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:00.296936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:00.297002 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:00.297315 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.297820 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.298504 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.298703 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.298754 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.301423 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:60399: BlockingRecv error: recv got EOF from 127.27.68.65:60399 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:00.440116 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:00.435652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:37647)
0929 03:23:00.435790 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:00.435805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:00.435902 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:00.436058 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:00.436059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:00.436223 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:00.436275 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.436277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.436912 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.436913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.437429 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.437431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.437767 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.437768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.437808 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.438251 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:00.438255 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:00.440013 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260929 03:23:00.440168 4650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:00.435739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:37647 (local address 127.27.68.126:34337)
0929 03:23:00.435855 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:00.435857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:00.435879 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:23:00.435970 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:00.435971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:00.435983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:00.436036 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:00.436341 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.436833 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.437507 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.437699 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.437750 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.440102 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:37647: BlockingRecv error: recv got EOF from 127.27.68.66:37647 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:00 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:00.572109 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:00.567588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:50887)
0929 03:23:00.567716 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:00.567728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:00.567785 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:00.567967 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:00.567967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:00.568129 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:00.568178 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.568179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.568834 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.568835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.569362 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:00.569364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.569768 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.569769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:00.569810 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.570251 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:00.570256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:00.572050 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260929 03:23:00.572275 4650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:00.567777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:50887 (local address 127.27.68.126:34337)
0929 03:23:00.567842 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:00.567844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:00.567853 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:00.567875 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:00.567876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:00.567888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:00.567954 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:00.568255 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.568754 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.569469 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:00.569698 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:00.569765 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:00.572176 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:50887: BlockingRecv error: recv got EOF from 127.27.68.67:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:01.306939 4653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:01.302062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:45015)
0929 03:23:01.302296 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:01.302314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:01.302392 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:01.302590 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:01.302591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:01.302756 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:01.302808 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.302810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.303468 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.303468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.303990 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.303992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.304343 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.304343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.304384 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.304847 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:01.304851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:01.306880 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
W20260929 03:23:01.307073 4654 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:01.302286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:45015 (local address 127.27.68.126:34337)
0929 03:23:01.302455 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:01.302458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:01.302468 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:23:01.302493 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:01.302493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:01.302504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:01.302572 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:01.302878 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.303403 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.304079 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.304276 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.304331 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.306980 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:45015: BlockingRecv error: recv got EOF from 127.27.68.65:45015 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:01.445592 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:01.440673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:33505)
0929 03:23:01.440964 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:01.440977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:01.441056 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:01.441231 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:01.441232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:01.441430 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:01.441492 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.441494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.442319 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.442320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.442873 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.442875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.443270 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.443271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.443335 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.443765 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:01.443769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:01.445528 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
W20260929 03:23:01.445670 4654 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:01.440789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:33505 (local address 127.27.68.126:34337)
0929 03:23:01.440968 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:01.440982 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:01.441036 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:23:01.441143 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:01.441144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:01.441156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:01.441226 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:01.441576 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.442225 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.442957 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.443155 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.443250 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.445622 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:33505: BlockingRecv error: recv got EOF from 127.27.68.66:33505 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:01 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:01.577109 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:01.572603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:37297)
0929 03:23:01.572748 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:01.572758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:01.572819 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:01.572957 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:01.572957 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:01.573114 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:01.573173 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.573175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.573814 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.573815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.574338 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:01.574339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.574672 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.574673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:01.574712 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.575147 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:01.575152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:01.577012 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
W20260929 03:23:01.577174 4654 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:01.572695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:37297 (local address 127.27.68.126:34337)
0929 03:23:01.572749 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:01.572751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:01.572807 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:23:01.572882 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:01.572883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:01.572894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:01.572938 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:01.573244 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.573738 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.574419 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:01.574604 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:01.574659 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:01.577108 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:37297: BlockingRecv error: recv got EOF from 127.27.68.67:37297 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:02.312239 4657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:02.307635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:51151)
0929 03:23:02.307755 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:02.307764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:02.307806 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:02.308006 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:02.308007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:02.308165 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:02.308215 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.308217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.308855 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.308856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.309385 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.309386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.309710 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.309711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.309748 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.310186 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:02.310191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:02.312122 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
W20260929 03:23:02.312304 4658 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:02.307717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:51151 (local address 127.27.68.126:34337)
0929 03:23:02.307852 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:02.307855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:02.307864 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:02.307897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:02.307898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:02.307909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:02.307978 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:02.308279 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.308780 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.309456 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.309647 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.309702 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.312228 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:51151: BlockingRecv error: recv got EOF from 127.27.68.65:51151 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:02.451483 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:02.446406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:32935)
0929 03:23:02.446625 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:02.446636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:02.446728 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:02.446939 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:02.446940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:02.447190 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:02.447286 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.447289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.448049 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.448050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.448583 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.448585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.448934 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.448935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.448974 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.449422 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:02.449427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:02.451408 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
W20260929 03:23:02.451659 4658 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:02.446560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:32935 (local address 127.27.68.126:34337)
0929 03:23:02.446645 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:02.446647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:02.446696 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:23:02.446798 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:02.446799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:02.446830 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:02.446901 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:02.447370 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.447970 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.448667 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.448866 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.448924 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.451557 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:32935: BlockingRecv error: recv got EOF from 127.27.68.66:32935 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:02 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:02.582544 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:02.577658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:51971)
0929 03:23:02.577823 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:02.577833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:02.577931 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:02.578094 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:02.578095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:02.578288 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:02.578335 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.578337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.579017 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.579018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.579591 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:02.579593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.579980 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.579981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:02.580044 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.580568 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:02.580573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:02.582443 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
W20260929 03:23:02.582600 4658 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:02.577728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:51971 (local address 127.27.68.126:34337)
0929 03:23:02.577773 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:02.577776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:02.577911 (+ 135us) server_negotiation.cc:407] Connection header received
0929 03:23:02.578019 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:02.578020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:02.578034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:02.578078 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:02.578422 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.578925 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.579694 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:02.579889 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:02.579979 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:02.582532 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:51971: BlockingRecv error: recv got EOF from 127.27.68.67:51971 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:03.317780 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:03.312948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:38577)
0929 03:23:03.313073 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:03.313085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:03.313147 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:03.313368 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:03.313370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:03.313556 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:03.313641 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.313643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.314378 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.314379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.314915 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.314916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.315360 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.315360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.315405 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.315942 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:03.315947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:03.317721 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
W20260929 03:23:03.317919 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:03.313000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38577 (local address 127.27.68.126:34337)
0929 03:23:03.313134 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:03.313137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:03.313147 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:23:03.313256 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:03.313257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:03.313272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:03.313349 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:03.313723 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.314305 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.314999 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.315191 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.315283 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.317825 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:38577: BlockingRecv error: recv got EOF from 127.27.68.65:38577 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:03.462282 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:03.452140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:45291)
0929 03:23:03.452317 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:03.452327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:03.452382 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:03.452531 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:03.452531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:03.452692 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:03.452740 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.452742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.453375 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.453375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.453892 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.453893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.454233 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.454233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.454272 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.460353 (+ 6081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:03.460359 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:03.462221 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
W20260929 03:23:03.462458 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:03.452221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:45291 (local address 127.27.68.126:34337)
0929 03:23:03.452294 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:03.452298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:03.452386 (+ 88us) server_negotiation.cc:407] Connection header received
0929 03:23:03.452434 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:03.452434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:03.452445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:03.452507 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:03.452806 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.453303 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.453970 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.454165 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.454219 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.462378 (+ 8159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:45291: BlockingRecv error: recv got EOF from 127.27.68.66:45291 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:03 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:03.588347 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:03.583287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:53961)
0929 03:23:03.583446 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:03.583460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:03.583542 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:03.583735 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:03.583737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:03.583914 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:03.583961 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.583962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.584650 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.584651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.585204 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:03.585206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.585636 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.585636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:03.585738 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.586199 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:03.586205 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:03.588236 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
W20260929 03:23:03.588392 4662 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:03.583497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:53961 (local address 127.27.68.126:34337)
0929 03:23:03.583567 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:03.583570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:03.583611 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:23:03.583637 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:03.583637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:03.583650 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:03.583726 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:03.584049 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.584550 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.585299 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:03.585557 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:03.585637 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:03.588324 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:53961: BlockingRecv error: recv got EOF from 127.27.68.67:53961 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:04.323410 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:04.318288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:54991)
0929 03:23:04.318457 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:04.318467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:04.318568 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:04.318739 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:04.318740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:04.318905 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:04.318957 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.318959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.319721 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.319722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.320265 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.320266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.320597 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.320597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.320663 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.321118 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:04.321122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:04.323309 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
W20260929 03:23:04.323475 4666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:04.318380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:54991 (local address 127.27.68.126:34337)
0929 03:23:04.318486 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:04.318488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:04.318615 (+ 127us) server_negotiation.cc:407] Connection header received
0929 03:23:04.318646 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:04.318647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:04.318659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:04.318723 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:04.319029 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.319636 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.320338 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.320534 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.320584 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.323402 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:54991: BlockingRecv error: recv got EOF from 127.27.68.65:54991 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:04.467742 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:04.462876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:39463)
0929 03:23:04.463181 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:04.463194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:04.463307 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:04.463468 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:04.463469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:04.463665 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:04.463709 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.463712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.464407 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.464408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.464921 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.464923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.465317 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.465318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.465376 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.465867 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:04.465871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:04.467672 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":219,"threads_started":1}
W20260929 03:23:04.467852 4666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:04.463087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:39463 (local address 127.27.68.126:34337)
0929 03:23:04.463169 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:04.463171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:04.463289 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:23:04.463381 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:04.463382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:04.463402 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:04.463462 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:04.463804 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.464307 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.465030 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.465232 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.465317 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.467779 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:39463: BlockingRecv error: recv got EOF from 127.27.68.66:39463 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:04 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:04.593984 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:04.588876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:44451)
0929 03:23:04.589068 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:04.589081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:04.589184 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:04.589375 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:04.589376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:04.589575 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:04.589622 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.589624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.590391 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.590392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.591055 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:04.591058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.591504 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.591505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:04.591568 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.592101 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:04.592106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:04.593912 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
W20260929 03:23:04.594107 4666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:04.588965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:44451 (local address 127.27.68.126:34337)
0929 03:23:04.589069 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:04.589071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:04.589162 (+ 91us) server_negotiation.cc:407] Connection header received
0929 03:23:04.589276 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:04.589277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:04.589300 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:04.589366 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:04.589728 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.590298 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.591155 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:04.591426 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:04.591503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:04.594027 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:44451: BlockingRecv error: recv got EOF from 127.27.68.67:44451 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:05.328753 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:05.323975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:46771)
0929 03:23:05.324120 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:05.324131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:05.324181 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:05.324352 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:05.324353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:05.324518 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:05.324584 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.324586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.325237 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.325238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.325842 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.325844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.326228 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.326229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.326278 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.326739 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:05.326744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:05.328696 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260929 03:23:05.328883 4670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:05.324079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46771 (local address 127.27.68.126:34337)
0929 03:23:05.324212 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:05.324214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:05.324222 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:23:05.324247 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:05.324248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:05.324258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:05.324334 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:05.324653 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.325162 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.325950 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.326157 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.326214 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.328794 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:46771: BlockingRecv error: recv got EOF from 127.27.68.65:46771 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:05.472970 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:05.468419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:44913)
0929 03:23:05.468579 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:05.468597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:05.468654 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:05.468796 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:05.468796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:05.468960 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:05.469005 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.469007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.469652 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.469653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.470158 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.470159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.470558 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.470559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.470624 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.471136 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:05.471141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:05.472911 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
W20260929 03:23:05.473121 4670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:05.468530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:44913 (local address 127.27.68.126:34337)
0929 03:23:05.468594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:05.468598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:05.468641 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:23:05.468703 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:05.468704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:05.468716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:05.468776 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:05.469076 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.469593 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.470236 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.470474 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.470530 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.473041 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:44913: BlockingRecv error: recv got EOF from 127.27.68.66:44913 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:05 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:05.599516 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:05.594543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:45315)
0929 03:23:05.594723 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:05.594738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:05.594848 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:05.595028 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:05.595029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:05.595258 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:05.595304 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.595306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.596007 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.596008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.596561 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:05.596562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.596953 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.596954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:05.597020 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.597549 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:05.597554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:05.599407 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
W20260929 03:23:05.599560 4670 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:05.594645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45315 (local address 127.27.68.126:34337)
0929 03:23:05.594746 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:05.594748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:05.594828 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:23:05.594939 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:05.594940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:05.594955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:05.595007 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:05.595397 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.595909 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.596657 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:05.596864 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:05.596952 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:05.599493 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:45315: BlockingRecv error: recv got EOF from 127.27.68.67:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:06.333838 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:06.329303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:60295)
0929 03:23:06.329458 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:06.329472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:06.329552 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:06.329690 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:06.329691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:06.329856 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:06.329909 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.329910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.330552 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.330553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.331067 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.331068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.331415 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.331416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.331457 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.331898 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:06.331902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:06.333779 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
W20260929 03:23:06.334015 4674 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:06.329400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:60295 (local address 127.27.68.126:34337)
0929 03:23:06.329538 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:06.329539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:06.329548 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:06.329600 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:06.329600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:06.329610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:06.329669 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:06.329975 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.330479 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.331146 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.331354 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.331405 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.333920 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:60295: BlockingRecv error: recv got EOF from 127.27.68.65:60295 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:06.478466 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:06.473549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:47947)
0929 03:23:06.473729 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:06.473741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:06.473797 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:06.473952 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:06.473953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:06.474120 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:06.474167 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.474169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.474813 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.474813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.475448 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.475450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.475786 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.475787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.475825 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.476292 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:06.476296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:06.478353 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
W20260929 03:23:06.478514 4674 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:06.473739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:47947 (local address 127.27.68.126:34337)
0929 03:23:06.473792 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:06.473794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:06.473803 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:06.473858 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:06.473858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:06.473868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:06.473928 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:06.474241 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.474737 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.475528 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.475718 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.475774 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.478450 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:47947: BlockingRecv error: recv got EOF from 127.27.68.66:47947 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:06 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:06.604951 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:06.600107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:40143)
0929 03:23:06.600307 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:06.600319 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:06.600390 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:06.600540 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:06.600542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:06.600731 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:06.600777 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.600778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.601446 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.601448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.601986 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:06.601988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.602361 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.602362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:06.602418 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.602981 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:06.602986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:06.604854 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":117,"threads_started":1}
W20260929 03:23:06.605024 4674 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:06.600230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:40143 (local address 127.27.68.126:34337)
0929 03:23:06.600301 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:06.600304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:06.600370 (+ 66us) server_negotiation.cc:407] Connection header received
0929 03:23:06.600445 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:06.600446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:06.600460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:06.600533 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:06.600858 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.601363 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.602083 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:06.602275 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:06.602360 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:06.604949 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:40143: BlockingRecv error: recv got EOF from 127.27.68.67:40143 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:07.339180 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:07.334355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:37475)
0929 03:23:07.334502 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:07.334513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:07.334601 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:07.334757 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:07.334758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:07.334972 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:07.335022 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.335024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.335781 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.335781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.336300 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.336301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.336708 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.336709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.336761 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.337241 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:07.337245 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:07.339119 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260929 03:23:07.339365 4678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:07.334404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:37475 (local address 127.27.68.126:34337)
0929 03:23:07.334539 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:07.334541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:07.334582 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:23:07.334670 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:07.334670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:07.334684 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:07.334753 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:07.335097 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.335702 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.336400 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.336637 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.336690 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.339272 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:37475: BlockingRecv error: recv got EOF from 127.27.68.65:37475 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:07.483495 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:07.479020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:38821)
0929 03:23:07.479172 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:07.479183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:07.479275 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:07.479407 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:07.479408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:07.479574 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:07.479643 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.479644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.480266 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.480267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.480791 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.480792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.481107 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.481108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.481159 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.481582 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:07.481586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:07.483397 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260929 03:23:07.483544 4678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:07.479097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:38821 (local address 127.27.68.126:34337)
0929 03:23:07.479165 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:07.479167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:07.479268 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:23:07.479332 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:07.479333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:07.479343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:07.479384 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:07.479712 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.480198 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.480864 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.481045 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.481095 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.483480 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:38821: BlockingRecv error: recv got EOF from 127.27.68.66:38821 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:07 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.67@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:07.610241 4680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:07.605599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.67:44475)
0929 03:23:07.605750 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:07.605763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:07.605820 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:07.605976 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:07.605977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:07.606141 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:07.606189 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.606190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.606819 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.606820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.607628 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:07.607631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.608010 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.608011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:07.608049 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.608515 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:07.608520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:07.610182 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
W20260929 03:23:07.610366 4678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:07.605760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:44475 (local address 127.27.68.126:34337)
0929 03:23:07.605831 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:07.605836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:07.605847 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:07.605874 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:07.605875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:07.605888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:07.605951 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:07.606253 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.606754 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.607737 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:07.607946 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:07.608003 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:07.610282 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.67:44475: BlockingRecv error: recv got EOF from 127.27.68.67:44475 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:23:08 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:08 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.65@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:08.345046 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.339930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.65:44457)
0929 03:23:08.340084 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:08.340095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:08.340151 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:08.340337 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:08.340338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:08.340549 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:08.340611 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.340613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.341608 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.341609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.342139 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.342141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.342470 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.342471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.342509 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.342945 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:08.342949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:08.344945 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
W20260929 03:23:08.345117 4682 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:08.340004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:44457 (local address 127.27.68.126:34337)
0929 03:23:08.340155 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:08.340157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:08.340193 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:23:08.340224 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:08.340224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:08.340237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:08.340327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:08.340690 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.341532 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.342214 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.342405 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.342455 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.345030 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.65:44457: BlockingRecv error: recv got EOF from 127.27.68.65:44457 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
Sep 29 03:23:08 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:23:08 dist-test-slave-fzwk krb5kdc[3623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.68.66@KRBTEST.COM for oryx/127.27.68.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:23:08.492488 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.484102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34337 (local address 127.27.68.66:43115)
0929 03:23:08.484259 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:08.484271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:08.484341 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:08.484473 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:08.484474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:08.484637 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:08.484687 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.484688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.485346 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.485347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.486163 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.486167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.486543 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.486544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.486616 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.490500 (+ 3884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:08.490506 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:08.492412 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.68.126:34337: Server oryx/127.27.68.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260929 03:23:08.492643 4682 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:08.484206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:43115 (local address 127.27.68.126:34337)
0929 03:23:08.484278 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:08.484280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:08.484324 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:23:08.484392 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:08.484392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:08.484403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:08.484454 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:08.484755 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.485258 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.486246 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.486467 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.486546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.492552 (+ 6006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.68.66:43115: BlockingRecv error: recv got EOF from 127.27.68.66:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:23:08.529187 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4206
I20260929 03:23:08.534708 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4358
I20260929 03:23:08.541216 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4492
I20260929 03:23:08.545964 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4136
2026-09-29T03:23:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17497 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:23:08Z Disabled control of system clock
I20260929 03:23:08.574331 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43931
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:43931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:23:08.656458 4687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:08.656733 4687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:08.656769 4687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:08.658017 4687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:23:08.658098 4687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:08.658119 4687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:08.658140 4687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:23:08.658169 4687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:23:08.659687 4687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:43931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:08.659998 4687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:08.660323 4687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:08.665271 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:08.665351 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:08.665449 4687 server_base.cc:1034] running on GCE node
W20260929 03:23:08.665268 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:08.665795 4687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:08.666143 4687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:08.667296 4687 hybrid_clock.cc:648] HybridClock initialized: now 1790652188667283 us; error 39 us; skew 500 ppm
I20260929 03:23:08.668566 4687 webserver.cc:460] Webserver started at http://127.27.68.126:34155/ using document root <none> and password file <none>
I20260929 03:23:08.668834 4687 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:08.668884 4687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:08.669052 4687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:08.670158 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "7fe9a5baa3c5488faf5cb5905733a2d2"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "4d5e6dab4f14a7d9404957217648e3d3"
server_key_iv: "6e21e50fd694ddf24c8316fa7db86ea7"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.670552 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "7fe9a5baa3c5488faf5cb5905733a2d2"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "4d5e6dab4f14a7d9404957217648e3d3"
server_key_iv: "6e21e50fd694ddf24c8316fa7db86ea7"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.672006 4687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:23:08.672770 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.672952 4687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:23:08.673049 4687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "7fe9a5baa3c5488faf5cb5905733a2d2"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "4d5e6dab4f14a7d9404957217648e3d3"
server_key_iv: "6e21e50fd694ddf24c8316fa7db86ea7"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.673130 4687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:08.683388 4687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:08.683712 4687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:08.683868 4687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:08.688043 4687 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:43931
I20260929 03:23:08.688112 4753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:43931 every 8 connection(s)
I20260929 03:23:08.688449 4687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:23:08.688799 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4687
I20260929 03:23:08.688905 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:23:08.689116 27921 external_mini_cluster.cc:1428] Setting key 67744781653e8df36a637d0b5c62c9f9
I20260929 03:23:08.689242 4754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:08.692477 4754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2: Bootstrap starting.
I20260929 03:23:08.693337 4754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:08.693665 4754 log.cc:824] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:08.694469 4754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2: No bootstrap required, opened a new log
I20260929 03:23:08.695037 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.689688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.27.68.126:43931)
0929 03:23:08.689933 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:08.689940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:08.689966 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:23:08.690391 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:08.690397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:08.690466 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:08.690638 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:08.690865 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.691831 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.692679 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.693360 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.693523 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.694275 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:08.694279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:08.694279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:08.694495 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:08.694538 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:08.694553 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:08.694638 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:08.694735 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:23:08.694820 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260929 03:23:08.696518 4754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } }
I20260929 03:23:08.696663 4754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:08.696691 4754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe9a5baa3c5488faf5cb5905733a2d2, State: Initialized, Role: FOLLOWER
I20260929 03:23:08.696846 4754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } }
I20260929 03:23:08.696930 4754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:23:08.696985 4754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:23:08.697036 4754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:08.697630 4754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } }
I20260929 03:23:08.697753 4754 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe9a5baa3c5488faf5cb5905733a2d2; no voters:
I20260929 03:23:08.697953 4754 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:23:08.697990 4759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:08.698185 4759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe9a5baa3c5488faf5cb5905733a2d2, State: Running, Role: LEADER
I20260929 03:23:08.698259 4754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:23:08.698318 4759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } }
I20260929 03:23:08.698699 4761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe9a5baa3c5488faf5cb5905733a2d2. Latest consensus state: current_term: 1 leader_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } } }
I20260929 03:23:08.698694 4760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9a5baa3c5488faf5cb5905733a2d2" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 43931 } } }
I20260929 03:23:08.698861 4760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:08.699049 4767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:23:08.699059 4761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:08.699601 4767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:23:08.701242 4767 catalog_manager.cc:1347] Generated new cluster ID: b59da8ba39cb499784158babc81ed3f9
I20260929 03:23:08.701341 4767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:23:08.711196 4767 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:23:08.711768 4767 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:23:08.713550 4767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe9a5baa3c5488faf5cb5905733a2d2: Generated new TSK 0
I20260929 03:23:08.713739 4767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:23:08.716678 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--builtin_ntp_servers=127.27.68.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:23:08.801410 4778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:08.801668 4778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:08.801705 4778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:08.802894 4778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:08.802964 4778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:08.803009 4778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:23:08.804459 4778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:08.804738 4778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:08.805048 4778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:08.811004 4783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:08.811045 4784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:08.811123 4786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:08.811326 4778 server_base.cc:1034] running on GCE node
I20260929 03:23:08.811594 4778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:08.811864 4778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:08.813037 4778 hybrid_clock.cc:648] HybridClock initialized: now 1790652188813021 us; error 32 us; skew 500 ppm
I20260929 03:23:08.814456 4778 webserver.cc:460] Webserver started at http://127.27.68.65:39557/ using document root <none> and password file <none>
I20260929 03:23:08.814723 4778 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:08.814793 4778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:08.814968 4778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:08.816152 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "b68a30847dea47028d2cb27d6dc7ef96"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "93fd70af12b4befc547050ba85a8bd9c"
server_key_iv: "b6ca4999da6895e553fe78f71f34c832"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.816545 4778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "b68a30847dea47028d2cb27d6dc7ef96"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "93fd70af12b4befc547050ba85a8bd9c"
server_key_iv: "b6ca4999da6895e553fe78f71f34c832"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.817940 4778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:23:08.818742 4792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.818971 4778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:08.819072 4778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "b68a30847dea47028d2cb27d6dc7ef96"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "93fd70af12b4befc547050ba85a8bd9c"
server_key_iv: "b6ca4999da6895e553fe78f71f34c832"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.819142 4778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:08.837980 4778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:08.838284 4778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:08.838394 4778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:08.838594 4778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:08.838958 4778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:08.838996 4778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.839043 4778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:08.839071 4778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.844950 4778 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:43411
I20260929 03:23:08.845036 4905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:43411 every 8 connection(s)
I20260929 03:23:08.845386 4778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:23:08.850544 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.846208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34787 (local address 127.27.68.126:43931)
0929 03:23:08.846270 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:08.846273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:08.846504 (+ 231us) server_negotiation.cc:407] Connection header received
0929 03:23:08.846988 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:08.846989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:08.847000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:08.847065 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:08.847792 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.848288 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.849256 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.849440 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.849500 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.850296 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:08.850297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:08.850297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:08.850301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:08.850315 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:08.850319 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:08.850361 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:08.850443 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:23:08.850480 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:23:08.850723 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.846170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43931 (local address 127.27.68.65:34787)
0929 03:23:08.846375 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:08.846417 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:08.846900 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:08.847119 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:08.847123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:08.847131 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:08.847710 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.847716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.848376 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.848376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.849167 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.849169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.849521 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.849522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.849636 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.850153 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:08.850154 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:08.850190 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:08.850213 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:08.850371 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:08.850375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:08.850378 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:08.850431 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:23:08.850516 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260929 03:23:08.851348 4906 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43931
I20260929 03:23:08.851460 4906 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:08.851648 4906 heartbeater.cc:511] Master 127.27.68.126:43931 requested a full tablet report, sending...
I20260929 03:23:08.852164 4718 ts_manager.cc:194] Registered new tserver with Master: b68a30847dea47028d2cb27d6dc7ef96 (127.27.68.65:43411)
I20260929 03:23:08.852447 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4778
I20260929 03:23:08.852546 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:23:08.852734 27921 external_mini_cluster.cc:1428] Setting key b9d75a85389e94d67e5a7a90af8297b6
I20260929 03:23:08.852908 4718 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.65:34787
I20260929 03:23:08.853816 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--builtin_ntp_servers=127.27.68.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:23:08.936517 4909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:08.936762 4909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:08.936784 4909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:08.937885 4909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:08.937922 4909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:08.937947 4909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:23:08.939389 4909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:08.939610 4909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:08.939877 4909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:08.944849 4917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:08.944852 4914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:08.944849 4915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:08.945355 4909 server_base.cc:1034] running on GCE node
I20260929 03:23:08.945588 4909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:08.945811 4909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:08.946977 4909 hybrid_clock.cc:648] HybridClock initialized: now 1790652188946971 us; error 37 us; skew 500 ppm
I20260929 03:23:08.948261 4909 webserver.cc:460] Webserver started at http://127.27.68.66:43197/ using document root <none> and password file <none>
I20260929 03:23:08.948514 4909 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:08.948591 4909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:08.948751 4909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:08.949855 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "2a441123fca643a99e336329e7a68639"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "cd0f5746fcee5c7fe4dc2a0454c5968a"
server_key_iv: "4f8bc0e75cc66a30dc0c366e5fb3043f"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.950219 4909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "2a441123fca643a99e336329e7a68639"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "cd0f5746fcee5c7fe4dc2a0454c5968a"
server_key_iv: "4f8bc0e75cc66a30dc0c366e5fb3043f"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.951642 4909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:23:08.952450 4923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.952630 4909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:08.952742 4909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "2a441123fca643a99e336329e7a68639"
format_stamp: "Formatted at 2026-09-29 03:23:08 on dist-test-slave-fzwk"
server_key: "cd0f5746fcee5c7fe4dc2a0454c5968a"
server_key_iv: "4f8bc0e75cc66a30dc0c366e5fb3043f"
server_key_version: "encryptionkey@0"
I20260929 03:23:08.952821 4909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:08.980233 4909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:08.980595 4909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:08.980758 4909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:08.980995 4909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:08.981355 4909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:08.981410 4909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.981452 4909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:08.981478 4909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:08.987577 4909 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:43535
I20260929 03:23:08.987622 5036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:43535 every 8 connection(s)
I20260929 03:23:08.988035 4909 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:23:08.989157 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 4909
I20260929 03:23:08.989269 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:23:08.989450 27921 external_mini_cluster.cc:1428] Setting key e7257d6cd6c47655cef6002e7eefbca0
I20260929 03:23:08.990403 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--builtin_ntp_servers=127.27.68.84:33549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 03:23:08.994131 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.988828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:50863 (local address 127.27.68.126:43931)
0929 03:23:08.988888 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:08.988890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:08.989176 (+ 286us) server_negotiation.cc:407] Connection header received
0929 03:23:08.989915 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:08.989917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:08.989930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:08.989986 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:08.990828 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.991388 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.992424 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.992775 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.992846 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.993762 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:08.993764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:08.993764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:08.993770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:08.993787 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:08.993792 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:08.993853 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:08.993967 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:23:08.994005 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:23:08.994292 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:08.988824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43931 (local address 127.27.68.66:50863)
0929 03:23:08.989053 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:08.989088 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:08.989812 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:08.990076 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:08.990080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:08.990091 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:08.990732 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.990739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.991484 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.991485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.992325 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:08.992327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:08.992865 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:08.992866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:08.993003 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:08.993605 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:08.993605 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:08.993635 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:08.993660 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:08.993860 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:08.993863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:08.993866 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:08.993953 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:23:08.994067 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260929 03:23:08.994915 5037 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43931
I20260929 03:23:08.995039 5037 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:08.995309 5037 heartbeater.cc:511] Master 127.27.68.126:43931 requested a full tablet report, sending...
I20260929 03:23:08.995751 4717 ts_manager.cc:194] Registered new tserver with Master: 2a441123fca643a99e336329e7a68639 (127.27.68.66:43535)
I20260929 03:23:08.996148 4717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.66:50863
W20260929 03:23:09.074579 5040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:09.074837 5040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:09.074875 5040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:09.076246 5040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:09.076311 5040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:09.076354 5040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:23:09.077824 5040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:43931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:09.078104 5040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:09.078421 5040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:09.083518 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.083576 5040 server_base.cc:1034] running on GCE node
W20260929 03:23:09.083541 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.083518 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.084015 5040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:09.084332 5040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:09.085526 5040 hybrid_clock.cc:648] HybridClock initialized: now 1790652189085522 us; error 36 us; skew 500 ppm
I20260929 03:23:09.086984 5040 webserver.cc:460] Webserver started at http://127.27.68.67:46665/ using document root <none> and password file <none>
I20260929 03:23:09.087292 5040 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:09.087386 5040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:09.087527 5040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:09.088631 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "6db71ab532cd4e5e8f58b361e5c017f0"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "19549d479614c85f9f2f029bddc596b6"
server_key_iv: "741bbe9d91293bd20cce642a6ef97766"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.089010 5040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "6db71ab532cd4e5e8f58b361e5c017f0"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "19549d479614c85f9f2f029bddc596b6"
server_key_iv: "741bbe9d91293bd20cce642a6ef97766"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.090395 5040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:23:09.091130 5054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.091312 5040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:23:09.091426 5040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "6db71ab532cd4e5e8f58b361e5c017f0"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "19549d479614c85f9f2f029bddc596b6"
server_key_iv: "741bbe9d91293bd20cce642a6ef97766"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.091519 5040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:09.108044 5040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:09.108399 5040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:09.108564 5040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:09.108811 5040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:09.109179 5040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:09.109236 5040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.109279 5040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:09.109306 5040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.115262 5040 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:44107
I20260929 03:23:09.115291 5167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:44107 every 8 connection(s)
I20260929 03:23:09.115671 5040 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:23:09.116358 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5040
I20260929 03:23:09.116451 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:23:09.116665 27921 external_mini_cluster.cc:1428] Setting key 337eb76dbc3ee275b50528b1f7efbc9c
I20260929 03:23:09.121071 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.116428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45789 (local address 127.27.68.126:43931)
0929 03:23:09.116636 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.116638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.116745 (+ 107us) server_negotiation.cc:407] Connection header received
0929 03:23:09.117450 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.117451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.117463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.117523 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.118249 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:09.118736 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:09.119791 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:09.119984 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:09.120040 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:09.120835 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:09.120836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:09.120837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:09.120841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:09.120854 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:09.120858 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:09.120901 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:09.120987 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 03:23:09.121010 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260929 03:23:09.121259 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.116389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:43931 (local address 127.27.68.67:45789)
0929 03:23:09.116607 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:09.116657 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:09.117355 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:09.117599 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:09.117603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:09.117614 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:09.118162 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:09.118167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:09.118819 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:09.118820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:09.119718 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:09.119720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:09.120060 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:09.120061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:09.120173 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:09.120704 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:09.120705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:09.120734 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:09.120758 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:09.120914 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:09.120918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:09.120921 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:09.120974 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:23:09.121056 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260929 03:23:09.121776 5168 heartbeater.cc:347] Connected to a master server at 127.27.68.126:43931
I20260929 03:23:09.121860 5168 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:09.122005 5168 heartbeater.cc:511] Master 127.27.68.126:43931 requested a full tablet report, sending...
I20260929 03:23:09.122320 4717 ts_manager.cc:194] Registered new tserver with Master: 6db71ab532cd4e5e8f58b361e5c017f0 (127.27.68.67:44107)
I20260929 03:23:09.122651 4717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.67:45789
I20260929 03:23:09.128576 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:23:09.130517 4757 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:09.129985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.27.68.126:43931)
0929 03:23:09.130048 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.130050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.130165 (+ 115us) server_negotiation.cc:407] Connection header received
0929 03:23:09.130237 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.130237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.130246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.130300 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.130427 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33106: BlockingRecv error: recv got EOF from 127.0.0.1:33106 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:23:09.130620 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4778
I20260929 03:23:09.135855 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4909
I20260929 03:23:09.140415 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5040
I20260929 03:23:09.144814 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 4687
2026-09-29T03:23:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (597 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:23:09Z Disabled control of system clock
I20260929 03:23:09.171798 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42385
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:23:09.253332 5179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:09.253589 5179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:09.253611 5179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:09.254756 5179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:23:09.254801 5179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:09.254813 5179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:09.254827 5179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:23:09.254838 5179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:23:09.256192 5179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:09.256412 5179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:09.256659 5179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:09.261498 5185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.261502 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.261548 5184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.261837 5179 server_base.cc:1034] running on GCE node
I20260929 03:23:09.262037 5179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:09.262337 5179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:09.263476 5179 hybrid_clock.cc:648] HybridClock initialized: now 1790652189263465 us; error 29 us; skew 500 ppm
I20260929 03:23:09.264721 5179 webserver.cc:460] Webserver started at http://127.27.68.126:45619/ using document root <none> and password file <none>
I20260929 03:23:09.264986 5179 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:09.265034 5179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:09.265192 5179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:09.266323 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "d22b0ef86d3b4ea09a1f4186f96b4804"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "952ba73c7d372331d38e524bb7cab17f"
server_key_iv: "ace8067d702c1fba8d796e1c80011fa5"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.266707 5179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "d22b0ef86d3b4ea09a1f4186f96b4804"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "952ba73c7d372331d38e524bb7cab17f"
server_key_iv: "ace8067d702c1fba8d796e1c80011fa5"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.268584 5179 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260929 03:23:09.269315 5193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.269482 5179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:23:09.269580 5179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "d22b0ef86d3b4ea09a1f4186f96b4804"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "952ba73c7d372331d38e524bb7cab17f"
server_key_iv: "ace8067d702c1fba8d796e1c80011fa5"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.269683 5179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:09.273253 5179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:09.273550 5179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:09.273701 5179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:09.277710 5179 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42385
I20260929 03:23:09.277742 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42385 every 8 connection(s)
I20260929 03:23:09.278084 5179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:23:09.278746 5246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:09.281502 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804: Bootstrap starting.
I20260929 03:23:09.282023 5246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:09.282264 5246 log.cc:824] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:09.282891 5246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804: No bootstrap required, opened a new log
I20260929 03:23:09.284535 5246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } }
I20260929 03:23:09.284682 5246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:09.284704 5246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22b0ef86d3b4ea09a1f4186f96b4804, State: Initialized, Role: FOLLOWER
I20260929 03:23:09.284843 5246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } }
I20260929 03:23:09.284922 5246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:23:09.284946 5246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:23:09.284976 5246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:09.285499 5246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } }
I20260929 03:23:09.285588 5246 leader_election.cc:302] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d22b0ef86d3b4ea09a1f4186f96b4804; no voters:
I20260929 03:23:09.285712 5246 leader_election.cc:288] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:23:09.285797 5249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:09.285987 5246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:23:09.286067 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5179
I20260929 03:23:09.286167 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:23:09.286026 5249 raft_consensus.cc:686] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [term 1 LEADER]: Becoming Leader. State: Replica: d22b0ef86d3b4ea09a1f4186f96b4804, State: Running, Role: LEADER
I20260929 03:23:09.286255 5249 consensus_queue.cc:237] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } }
I20260929 03:23:09.286356 27921 external_mini_cluster.cc:1428] Setting key bf018d16571d091bf9a478619de09b55
I20260929 03:23:09.286811 5250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } } }
I20260929 03:23:09.286890 5250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:09.287469 5251 sys_catalog.cc:455] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d22b0ef86d3b4ea09a1f4186f96b4804. Latest consensus state: current_term: 1 leader_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d22b0ef86d3b4ea09a1f4186f96b4804" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42385 } } }
I20260929 03:23:09.287542 5251 sys_catalog.cc:458] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:09.287545 5259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:23:09.288293 5259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:23:09.289667 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.287194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.27.68.126:42385)
0929 03:23:09.287529 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.287539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.287778 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:23:09.287905 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.287913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.287963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.288080 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.288903 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:09.288907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:09.288907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:09.288912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:09.288983 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:09.289000 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:09.289068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:09.289204 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 03:23:09.289262 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":68,"threads_started":1}
I20260929 03:23:09.290097 5259 catalog_manager.cc:1347] Generated new cluster ID: 0db4c7d292544c7fa97740958c003ed5
I20260929 03:23:09.290158 5259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:23:09.329607 5259 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:23:09.329846 5259 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:23:09.334648 5259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d22b0ef86d3b4ea09a1f4186f96b4804: Generated new TSK 0
I20260929 03:23:09.334873 5259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:23:09.336781 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--builtin_ntp_servers=127.27.68.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:23:09.420215 5271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:09.420465 5271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:09.420506 5271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:09.421711 5271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:09.421778 5271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:09.421824 5271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:23:09.423286 5271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:09.423558 5271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:09.423866 5271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:09.429247 5276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.429513 5271 server_base.cc:1034] running on GCE node
W20260929 03:23:09.429247 5279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.429251 5277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.429843 5271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:09.430060 5271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:09.431278 5271 hybrid_clock.cc:648] HybridClock initialized: now 1790652189431262 us; error 39 us; skew 500 ppm
I20260929 03:23:09.432574 5271 webserver.cc:460] Webserver started at http://127.27.68.65:35915/ using document root <none> and password file <none>
I20260929 03:23:09.432827 5271 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:09.432898 5271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:09.433051 5271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:09.434142 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "c31fbad19bde4dc387f3c3e262a21a4e"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "04be8420096062ed0b72d18152ead933"
server_key_iv: "df38bd33dcff652b1f5e03823ee6f7b1"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.434523 5271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "c31fbad19bde4dc387f3c3e262a21a4e"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "04be8420096062ed0b72d18152ead933"
server_key_iv: "df38bd33dcff652b1f5e03823ee6f7b1"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.436025 5271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:23:09.436828 5285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.437006 5271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:09.437116 5271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "c31fbad19bde4dc387f3c3e262a21a4e"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "04be8420096062ed0b72d18152ead933"
server_key_iv: "df38bd33dcff652b1f5e03823ee6f7b1"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.437196 5271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:09.440749 5271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:09.441027 5271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:09.441169 5271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:09.441363 5271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:09.441672 5271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:09.441726 5271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.441766 5271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:09.441793 5271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.447660 5271 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:35885
I20260929 03:23:09.447707 5398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:35885 every 8 connection(s)
I20260929 03:23:09.448072 5271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:23:09.451429 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.448859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:39173 (local address 127.27.68.126:42385)
0929 03:23:09.448928 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.448931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.449184 (+ 253us) server_negotiation.cc:407] Connection header received
0929 03:23:09.449647 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.449648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.449656 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.449719 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.451103 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:09.451105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:09.451105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:09.451110 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:09.451129 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:09.451134 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:09.451206 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:09.451353 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:23:09.451370 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:23:09.451642 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.448862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42385 (local address 127.27.68.65:39173)
0929 03:23:09.449049 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:09.449092 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:09.449560 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:09.449775 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:09.449779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:09.449787 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:09.450917 (+ 1130us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:09.450917 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:09.450949 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:09.450983 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:09.451210 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:09.451273 (+ 63us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:09.451276 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:09.451360 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:23:09.451414 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260929 03:23:09.451952 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5271
I20260929 03:23:09.452066 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:23:09.452193 5399 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42385
I20260929 03:23:09.452294 27921 external_mini_cluster.cc:1428] Setting key 2e94ae0a234a48c72158fbab78c0f319
I20260929 03:23:09.452302 5399 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:09.452469 5399 heartbeater.cc:511] Master 127.27.68.126:42385 requested a full tablet report, sending...
I20260929 03:23:09.453082 5209 ts_manager.cc:194] Registered new tserver with Master: c31fbad19bde4dc387f3c3e262a21a4e (127.27.68.65:35885)
I20260929 03:23:09.453320 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--builtin_ntp_servers=127.27.68.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:23:09.535710 5402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:09.536003 5402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:09.536051 5402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:09.537310 5402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:09.537379 5402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:09.537420 5402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:23:09.539501 5402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:09.539996 5402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:09.540370 5402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:23:09.545807 5402 server_base.cc:1034] running on GCE node
W20260929 03:23:09.545794 5408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.545957 5410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.545794 5407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.546240 5402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:09.546509 5402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:09.547650 5402 hybrid_clock.cc:648] HybridClock initialized: now 1790652189547638 us; error 31 us; skew 500 ppm
I20260929 03:23:09.548978 5402 webserver.cc:460] Webserver started at http://127.27.68.66:36209/ using document root <none> and password file <none>
I20260929 03:23:09.549250 5402 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:09.549301 5402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:09.549474 5402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:09.550570 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "2386af0944a64f478f27cd1d2fa916bb"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "aad498a30e1f47b3b14e00bf10084890"
server_key_iv: "ff5fe78ff2ecd5c44548d3c02eb97633"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.550935 5402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "2386af0944a64f478f27cd1d2fa916bb"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "aad498a30e1f47b3b14e00bf10084890"
server_key_iv: "ff5fe78ff2ecd5c44548d3c02eb97633"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.552376 5402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:23:09.553162 5416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.553371 5402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:23:09.553473 5402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "2386af0944a64f478f27cd1d2fa916bb"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "aad498a30e1f47b3b14e00bf10084890"
server_key_iv: "ff5fe78ff2ecd5c44548d3c02eb97633"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.553563 5402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:09.557101 5402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:09.557384 5402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:09.557529 5402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:09.557734 5402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:09.558051 5402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:09.558106 5402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.558149 5402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:09.558183 5402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.564141 5402 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:41209
I20260929 03:23:09.564184 5529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:41209 every 8 connection(s)
I20260929 03:23:09.564550 5402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:23:09.567620 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.565346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:55137 (local address 127.27.68.126:42385)
0929 03:23:09.565422 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.565424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.565679 (+ 255us) server_negotiation.cc:407] Connection header received
0929 03:23:09.566141 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.566142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.566148 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.566223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.567424 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:09.567425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:09.567425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:09.567430 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:09.567448 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:09.567452 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:09.567503 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:09.567549 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:23:09.567562 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 03:23:09.567823 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.565328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42385 (local address 127.27.68.66:55137)
0929 03:23:09.565539 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:09.565582 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:09.566057 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:09.566280 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:09.566283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:09.566291 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:09.567197 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:09.567197 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:09.567274 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:09.567335 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:09.567500 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:09.567502 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:09.567505 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:09.567557 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:23:09.567608 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20260929 03:23:09.568166 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5402
I20260929 03:23:09.568295 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:23:09.568368 5530 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42385
I20260929 03:23:09.568470 5530 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:09.568486 27921 external_mini_cluster.cc:1428] Setting key 80feb28924356d999b642a953a2262ba
I20260929 03:23:09.568631 5530 heartbeater.cc:511] Master 127.27.68.126:42385 requested a full tablet report, sending...
I20260929 03:23:09.569048 5210 ts_manager.cc:194] Registered new tserver with Master: 2386af0944a64f478f27cd1d2fa916bb (127.27.68.66:41209)
I20260929 03:23:09.569581 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--builtin_ntp_servers=127.27.68.84:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:23:09.652181 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:09.652423 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:09.652446 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:09.653548 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:09.653584 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:09.653610 5533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:23:09.654877 5533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:09.655076 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:09.655405 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:09.660625 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:09.660620 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.660704 5533 server_base.cc:1034] running on GCE node
W20260929 03:23:09.660620 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:09.661036 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:09.661341 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:09.662518 5533 hybrid_clock.cc:648] HybridClock initialized: now 1790652189662501 us; error 28 us; skew 500 ppm
I20260929 03:23:09.663892 5533 webserver.cc:460] Webserver started at http://127.27.68.67:43007/ using document root <none> and password file <none>
I20260929 03:23:09.664160 5533 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:09.664232 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:09.664398 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:09.665674 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "b043ea0d7843463dafedcb8dc8fae9c3"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "1c804b9e298e4d3bcf6a3a97fffaace4"
server_key_iv: "7c5658568ab08f7a7b3045e094f87c4f"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.666069 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "b043ea0d7843463dafedcb8dc8fae9c3"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "1c804b9e298e4d3bcf6a3a97fffaace4"
server_key_iv: "7c5658568ab08f7a7b3045e094f87c4f"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.667515 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 03:23:09.668294 5547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.668473 5533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:23:09.668588 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "b043ea0d7843463dafedcb8dc8fae9c3"
format_stamp: "Formatted at 2026-09-29 03:23:09 on dist-test-slave-fzwk"
server_key: "1c804b9e298e4d3bcf6a3a97fffaace4"
server_key_iv: "7c5658568ab08f7a7b3045e094f87c4f"
server_key_version: "encryptionkey@0"
I20260929 03:23:09.668670 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:09.672353 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:09.672657 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:09.672807 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:09.673048 5533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:09.673359 5533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:09.673413 5533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.673456 5533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:09.673484 5533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:09.679447 5533 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:36139
I20260929 03:23:09.679502 5660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:36139 every 8 connection(s)
I20260929 03:23:09.679852 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:23:09.682919 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.680616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:56647 (local address 127.27.68.126:42385)
0929 03:23:09.680706 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.680708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.680902 (+ 194us) server_negotiation.cc:407] Connection header received
0929 03:23:09.681374 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.681375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.681381 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.681447 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.682677 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:09.682679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:09.682679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:23:09.682684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:09.682702 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:23:09.682707 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:09.682778 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:09.682831 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:23:09.682846 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:23:09.683097 5663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:09.680580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42385 (local address 127.27.68.67:56647)
0929 03:23:09.680781 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:09.680816 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:09.681278 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:09.681502 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:09.681505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:09.681513 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:09.682499 (+ 986us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:23:09.682500 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:09.682529 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:23:09.682560 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:09.682777 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:09.682779 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:09.682781 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:23:09.682830 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:23:09.682881 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260929 03:23:09.683715 5661 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42385
I20260929 03:23:09.683799 5661 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:09.683961 5661 heartbeater.cc:511] Master 127.27.68.126:42385 requested a full tablet report, sending...
I20260929 03:23:09.684382 5210 ts_manager.cc:194] Registered new tserver with Master: b043ea0d7843463dafedcb8dc8fae9c3 (127.27.68.67:36139)
I20260929 03:23:09.684540 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5533
I20260929 03:23:09.684646 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:23:09.684854 27921 external_mini_cluster.cc:1428] Setting key 36aa61b403a46711e54010bdd5d086ce
I20260929 03:23:09.685761 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:23:09.687870 5265 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:23:09.687349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.27.68.126:42385)
0929 03:23:09.687430 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:09.687432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:09.687518 (+ 86us) server_negotiation.cc:407] Connection header received
0929 03:23:09.687592 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:09.687594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:09.687600 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:09.687663 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:09.687787 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52686: BlockingRecv error: recv got EOF from 127.0.0.1:52686 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:23:09.687904 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5271
I20260929 03:23:09.692858 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5402
I20260929 03:23:09.696961 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5533
I20260929 03:23:09.701225 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5179
2026-09-29T03:23:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (556 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:23:09 dist-test-slave-fzwk krb5kdc[5671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:23:09 dist-test-slave-fzwk krb5kdc[5671](info): set up 2 sockets
Sep 29 03:23:09 dist-test-slave-fzwk krb5kdc[5671](info): commencing operation
krb5kdc: starting...
W20260929 03:23:10.744551 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:23:10 dist-test-slave-fzwk krb5kdc[5671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:23:10.862406 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:37701
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:37701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:10.948529 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:10.948787 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:10.948827 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:10.950066 5684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:23:10.950141 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:10.950170 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:10.950202 5684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:23:10.950227 5684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:23:10.951893 5684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:37701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:37701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:10.952201 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:10.952440 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:10.957357 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:10.957396 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:10.957414 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:10.957638 5684 server_base.cc:1034] running on GCE node
I20260929 03:23:10.957959 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:10.958250 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:10.959456 5684 hybrid_clock.cc:648] HybridClock initialized: now 1790652190959446 us; error 38 us; skew 500 ppm
Sep 29 03:23:10 dist-test-slave-fzwk krb5kdc[5671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652190, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:10.962116 5684 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:23:10.962633 5684 webserver.cc:460] Webserver started at http://127.27.68.126:44965/ using document root <none> and password file <none>
I20260929 03:23:10.962888 5684 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:10.962954 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:10.963114 5684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:10.964267 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "bcfb2234b00d4bd39597e6cfee110478"
format_stamp: "Formatted at 2026-09-29 03:23:10 on dist-test-slave-fzwk"
server_key: "fc470f59ed68bbcaf563cd9b97144631"
server_key_iv: "6cfb9f4a1ecaa39337492bdd0af8b956"
server_key_version: "encryptionkey@0"
I20260929 03:23:10.964638 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "bcfb2234b00d4bd39597e6cfee110478"
format_stamp: "Formatted at 2026-09-29 03:23:10 on dist-test-slave-fzwk"
server_key: "fc470f59ed68bbcaf563cd9b97144631"
server_key_iv: "6cfb9f4a1ecaa39337492bdd0af8b956"
server_key_version: "encryptionkey@0"
I20260929 03:23:10.966046 5684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:23:10.966768 5699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:10.966997 5684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:23:10.967123 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "bcfb2234b00d4bd39597e6cfee110478"
format_stamp: "Formatted at 2026-09-29 03:23:10 on dist-test-slave-fzwk"
server_key: "fc470f59ed68bbcaf563cd9b97144631"
server_key_iv: "6cfb9f4a1ecaa39337492bdd0af8b956"
server_key_version: "encryptionkey@0"
I20260929 03:23:10.967207 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:10.984103 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:10.986164 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:10.986340 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:10.990332 5684 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:37701
I20260929 03:23:10.990365 5751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:37701 every 8 connection(s)
I20260929 03:23:10.990775 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:23:10.991917 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:10.994308 5752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: Bootstrap starting.
I20260929 03:23:10.994936 5752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:10.995306 5752 log.cc:824] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:10.996038 5752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: No bootstrap required, opened a new log
I20260929 03:23:10.997542 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5684
I20260929 03:23:10.997651 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:23:10.997853 27921 external_mini_cluster.cc:1428] Setting key d66d2573c74291e0df49e7b1bd3e6c1b
I20260929 03:23:10.997948 5752 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } }
I20260929 03:23:10.998128 5752 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:10.998154 5752 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfb2234b00d4bd39597e6cfee110478, State: Initialized, Role: FOLLOWER
I20260929 03:23:10.998345 5752 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } }
I20260929 03:23:10.998438 5752 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:23:10.998508 5752 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:23:10.998611 5752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:10.999795 5752 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } }
I20260929 03:23:10.999907 5752 leader_election.cc:302] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcfb2234b00d4bd39597e6cfee110478; no voters:
I20260929 03:23:11.000069 5752 leader_election.cc:288] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:23:11.000146 5757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:11.000298 5757 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [term 1 LEADER]: Becoming Leader. State: Replica: bcfb2234b00d4bd39597e6cfee110478, State: Running, Role: LEADER
I20260929 03:23:11.000324 5752 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:23:11.000442 5757 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } }
I20260929 03:23:11.000775 5759 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcfb2234b00d4bd39597e6cfee110478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } } }
I20260929 03:23:11.000859 5759 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:11.001453 5764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:23:11.001710 5758 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcfb2234b00d4bd39597e6cfee110478. Latest consensus state: current_term: 1 leader_uuid: "bcfb2234b00d4bd39597e6cfee110478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcfb2234b00d4bd39597e6cfee110478" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37701 } } }
I20260929 03:23:11.001837 5758 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:11.001960 5764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:23:11.004562 5764 catalog_manager.cc:1347] Generated new cluster ID: 2b5557cc773c4b3895ce5805a5fd1f02
I20260929 03:23:11.004628 5764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:11.009793 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:10.998403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.27.68.126:37701)
0929 03:23:10.998604 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:10.998614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:10.998648 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:23:10.998782 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:10.998790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:10.998847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:10.999018 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:10.999669 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.001113 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.002265 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.002814 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.002961 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.005674 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:11.005679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:11.005680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:11.005693 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:11.007932 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.008390 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.008394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.008394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.008551 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.008795 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.008796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.008796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.009156 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:11.009332 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.009404 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.009569 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260929 03:23:11.013109 5764 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:23:11.013638 5764 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:23:11.022681 5764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: Generated new TSK 0
I20260929 03:23:11.022918 5764 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:23:11.054159 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:11.139976 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:11.140208 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:11.140235 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:11.141486 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:11.141537 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:11.141566 5778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:23:11.143142 5778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:11.143471 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:11.143726 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:11.148917 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:11.148934 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:11.149080 5778 server_base.cc:1034] running on GCE node
W20260929 03:23:11.148910 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:11.149384 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:11.149690 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:11.150857 5778 hybrid_clock.cc:648] HybridClock initialized: now 1790652191150845 us; error 44 us; skew 500 ppm
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:11.153501 5778 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:23:11.154003 5778 webserver.cc:460] Webserver started at http://127.27.68.65:37773/ using document root <none> and password file <none>
I20260929 03:23:11.154218 5778 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:11.154260 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:11.154353 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:11.155421 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "202b7d2d56434095b11de7e3f82d4ae2"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "950c1a4ba35bab65fc4a7e60c9307647"
server_key_iv: "b8b15093a3e2bc437810e1d620dfb0ba"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.155750 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "202b7d2d56434095b11de7e3f82d4ae2"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "950c1a4ba35bab65fc4a7e60c9307647"
server_key_iv: "b8b15093a3e2bc437810e1d620dfb0ba"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.157169 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:23:11.157853 5793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.158025 5778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:23:11.158100 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "202b7d2d56434095b11de7e3f82d4ae2"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "950c1a4ba35bab65fc4a7e60c9307647"
server_key_iv: "b8b15093a3e2bc437810e1d620dfb0ba"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.158155 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:11.175988 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:11.178527 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:11.178653 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:11.178849 5778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:11.179277 5778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:11.179319 5778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.179344 5778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:11.179358 5778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.185463 5778 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:44405
I20260929 03:23:11.185513 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:44405 every 8 connection(s)
I20260929 03:23:11.185859 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:23:11.189646 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5778
I20260929 03:23:11.189759 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:23:11.189952 27921 external_mini_cluster.cc:1428] Setting key bf2630618971814fd660544ae31a5c6d
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:11.196815 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.186634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38415 (local address 127.27.68.126:37701)
0929 03:23:11.186714 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.186717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.186947 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:23:11.187547 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.187548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.187558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.187623 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:11.188636 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.189152 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.190282 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.190582 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.190658 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.193695 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:11.193697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:11.193697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:11.193704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:11.195154 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.195763 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.195764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.195764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.195818 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.196135 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.196135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.196136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.196241 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:11.196290 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.196669 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.196747 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:23:11.197113 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.186606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37701 (local address 127.27.68.65:38415)
0929 03:23:11.186805 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.186852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.187444 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.187718 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.187722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.188333 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:11.188541 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.188546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.189239 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.189240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.190189 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.190192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.190691 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.190692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.190871 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.191512 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:11.191524 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:11.193586 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:11.195273 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.195284 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.195287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.195671 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.195891 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.195893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.195893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.196063 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.196308 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:11.196315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:11.196518 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.196641 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.196839 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":116,"threads_started":1}
I20260929 03:23:11.197750 5907 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37701
I20260929 03:23:11.197846 5907 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:11.198050 5907 heartbeater.cc:511] Master 127.27.68.126:37701 requested a full tablet report, sending...
I20260929 03:23:11.198621 5716 ts_manager.cc:194] Registered new tserver with Master: 202b7d2d56434095b11de7e3f82d4ae2 (127.27.68.65:44405)
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:23:11.199345 5716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:38415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:23:11.217965 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:11.301424 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:11.301662 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:11.301688 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:11.302933 5912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:11.302976 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:11.303004 5912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:23:11.304550 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:11.304760 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:11.304947 5912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:11.309923 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:11.309959 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:11.310046 5912 server_base.cc:1034] running on GCE node
W20260929 03:23:11.310148 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:11.310410 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:11.310659 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:11.311805 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790652191311784 us; error 43 us; skew 500 ppm
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:11.314520 5912 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:23:11.314960 5912 webserver.cc:460] Webserver started at http://127.27.68.66:36117/ using document root <none> and password file <none>
I20260929 03:23:11.315160 5912 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:11.315205 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:11.315388 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:11.316519 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "494de34d612a47d3b17059a8914c317e"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "06ef4bc345378ca2e2d36ff3f6c03670"
server_key_iv: "748faaddd344cbcebd896c92b8db1cc9"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.316859 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "494de34d612a47d3b17059a8914c317e"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "06ef4bc345378ca2e2d36ff3f6c03670"
server_key_iv: "748faaddd344cbcebd896c92b8db1cc9"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.318270 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:23:11.319063 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.319279 5912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:23:11.319370 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "494de34d612a47d3b17059a8914c317e"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "06ef4bc345378ca2e2d36ff3f6c03670"
server_key_iv: "748faaddd344cbcebd896c92b8db1cc9"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.319432 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:11.327628 5912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:11.329582 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:11.329710 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:11.329897 5912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:11.330222 5912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:11.330257 5912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.330281 5912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:11.330330 5912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.336436 5912 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:43669
I20260929 03:23:11.336525 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:43669 every 8 connection(s)
I20260929 03:23:11.336848 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:11.343295 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 5912
I20260929 03:23:11.343395 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:23:11.343586 27921 external_mini_cluster.cc:1428] Setting key 2cc561e96f1da688c8f945d9dcea1c5a
I20260929 03:23:11.346832 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.337653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:50021 (local address 127.27.68.126:37701)
0929 03:23:11.337726 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.337728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.337909 (+ 181us) server_negotiation.cc:407] Connection header received
0929 03:23:11.338395 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.338396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.338405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.338487 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:11.339493 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.340068 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.341048 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.341261 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.341310 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.343652 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:11.343654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:11.343655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:11.343664 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:11.345421 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.345837 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.345838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.345839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.345914 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.346216 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.346217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.346217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.346320 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:11.346371 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.346703 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.346769 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:23:11.347146 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.337633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37701 (local address 127.27.68.66:50021)
0929 03:23:11.337785 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.337826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.338313 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.338546 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.338549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.339136 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:11.339409 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.339415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.340162 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.340163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.340970 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.340972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.341334 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.341335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.341434 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.342030 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:11.342039 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:11.343561 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:11.345503 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.345510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.345511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.345735 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.345988 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.345989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.345990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.346149 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.346382 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:11.346387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:11.346560 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.346679 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.346864 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260929 03:23:11.347692 6041 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37701
I20260929 03:23:11.347785 6041 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:11.347990 6041 heartbeater.cc:511] Master 127.27.68.126:37701 requested a full tablet report, sending...
I20260929 03:23:11.348418 5716 ts_manager.cc:194] Registered new tserver with Master: 494de34d612a47d3b17059a8914c317e (127.27.68.66:43669)
I20260929 03:23:11.348996 5716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:50021
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:23:11.370028 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:11.452021 6046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:11.452252 6046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:11.452277 6046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:11.453435 6046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:11.453485 6046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:11.453511 6046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:23:11.454982 6046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:11.455278 6046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:11.455533 6046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:11.460618 6051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:11.460644 6054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:11.460618 6052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:11.460647 6046 server_base.cc:1034] running on GCE node
I20260929 03:23:11.460981 6046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:11.461231 6046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:11.462379 6046 hybrid_clock.cc:648] HybridClock initialized: now 1790652191462356 us; error 42 us; skew 500 ppm
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:11.465046 6046 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:23:11.465539 6046 webserver.cc:460] Webserver started at http://127.27.68.67:45713/ using document root <none> and password file <none>
I20260929 03:23:11.465780 6046 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:11.465826 6046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:11.465997 6046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:23:11.467089 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "b03b9621d41072c4c83997ae798ea5a0"
server_key_iv: "f9a1688068d0a9802c4dc80fb520f72e"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.467515 6046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "b03b9621d41072c4c83997ae798ea5a0"
server_key_iv: "f9a1688068d0a9802c4dc80fb520f72e"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.468942 6046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:11.469722 6061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.469898 6046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:11.469998 6046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0"
format_stamp: "Formatted at 2026-09-29 03:23:11 on dist-test-slave-fzwk"
server_key: "b03b9621d41072c4c83997ae798ea5a0"
server_key_iv: "f9a1688068d0a9802c4dc80fb520f72e"
server_key_version: "encryptionkey@0"
I20260929 03:23:11.470081 6046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:11.476725 6046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:11.479108 6046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:11.479302 6046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:11.479547 6046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:11.479872 6046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:11.479925 6046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.479969 6046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:11.479997 6046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:11.486119 6046 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:46229
I20260929 03:23:11.486167 6174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:46229 every 8 connection(s)
I20260929 03:23:11.486503 6046 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[5671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:11.495312 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 6046
I20260929 03:23:11.495425 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:23:11.495649 27921 external_mini_cluster.cc:1428] Setting key 9a11bc0bfe3a58eee213bd8453a48f8a
I20260929 03:23:11.496256 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.487309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:37693 (local address 127.27.68.126:37701)
0929 03:23:11.487393 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.487395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.487584 (+ 189us) server_negotiation.cc:407] Connection header received
0929 03:23:11.488074 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.488075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.488085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.488148 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:11.489113 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.489607 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.490600 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.490792 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.490847 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.493163 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:11.493165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:11.493165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:11.493173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:11.494644 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.495018 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.495018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.495019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.495071 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.495440 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.495441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.495441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.495582 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:11.495643 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.496045 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.496151 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:23:11.496505 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.487297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37701 (local address 127.27.68.67:37693)
0929 03:23:11.487448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.487492 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.487988 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.488206 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.488210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.488813 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:11.489022 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.489027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.489699 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.489700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.490514 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.490516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.490872 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.490872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.490978 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.491613 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:11.491623 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:11.493064 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:11.494724 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.494729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.494732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.494950 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.495165 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:11.495167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:11.495167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:11.495354 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:11.495658 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:11.495666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:11.495856 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.495975 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.496221 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260929 03:23:11.496937 6175 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37701
I20260929 03:23:11.497058 6175 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:11.497254 6175 heartbeater.cc:511] Master 127.27.68.126:37701 requested a full tablet report, sending...
I20260929 03:23:11.497668 5716 ts_manager.cc:194] Registered new tserver with Master: 4f5df2d49b9248dd94dbb74c0b1e7af0 (127.27.68.67:46229)
I20260929 03:23:11.498020 5716 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:37693
I20260929 03:23:11.507673 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:23:11.514777 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.509033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.27.68.126:37701)
0929 03:23:11.509093 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.509095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.509202 (+ 107us) server_negotiation.cc:407] Connection header received
0929 03:23:11.509246 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.509247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.509256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.509309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:11.509856 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.510342 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.511048 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.511306 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.511357 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.512622 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:11.512623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:11.512623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:11.512628 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:11.513826 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.514149 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.514149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.514150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.514208 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:11.514332 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:11.514333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:11.514333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:11.514476 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:11.514548 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.514646 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.514709 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:23:11.516641 5716 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:23:11.518275 5716 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:23:11.525254 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43669 (local address 127.0.0.1:44614)
0929 03:23:11.520714 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.520745 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.520812 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.521367 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.521370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.521374 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:11.521456 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.521458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522870 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.522870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.524147 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.524150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.524942 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.525008 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.525014 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.525066 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.525112 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":54,"thread_start_us":57,"threads_started":1}
I20260929 03:23:11.525290 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.27.68.65:44405)
0929 03:23:11.521223 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.521228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.521239 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:11.521280 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.521282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.521299 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.521374 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:11.521613 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522462 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.523977 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.524725 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524810 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.524876 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.525168 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.525216 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":104,"threads_started":1}
I20260929 03:23:11.525278 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:46229 (local address 127.0.0.1:60666)
0929 03:23:11.520974 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.520982 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.521015 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.521369 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.521370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.521372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:11.521456 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.521459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522875 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.522877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.524148 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.524150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.525047 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.525048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.525107 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.525112 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.525167 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.525203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":92,"thread_start_us":48,"threads_started":1}
I20260929 03:23:11.525586 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.27.68.66:43669)
0929 03:23:11.521174 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.521180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.521191 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:11.521234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.521236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.521252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.521535 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:11.521622 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522742 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524255 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.524832 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524931 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.524991 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.525417 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.525497 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260929 03:23:11.525916 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44405 (local address 127.0.0.1:43478)
0929 03:23:11.520698 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.520745 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.520812 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.521463 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.521464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.521466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:11.521524 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.521525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522565 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.522566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.523880 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.523882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.524827 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.524931 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.524950 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.525020 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.525083 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":57,"thread_start_us":102,"threads_started":1}
I20260929 03:23:11.526023 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.520977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.27.68.67:46229)
0929 03:23:11.521176 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.521180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.521191 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:11.521234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.521236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.521252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.521535 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:11.521599 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.522785 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.524252 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.524977 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.525050 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.525103 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.525197 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.525241 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260929 03:23:11.526417 5975 tablet_service.cc:1467] Processing CreateTablet for tablet a95809cf5916449ba5eb0e120f33eadb (DEFAULT_TABLE table=test-table [id=d4eb6c29edd4496083fa226abd143d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:11.526624 5841 tablet_service.cc:1467] Processing CreateTablet for tablet a95809cf5916449ba5eb0e120f33eadb (DEFAULT_TABLE table=test-table [id=d4eb6c29edd4496083fa226abd143d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:11.526832 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95809cf5916449ba5eb0e120f33eadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:11.526932 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95809cf5916449ba5eb0e120f33eadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:11.527604 6109 tablet_service.cc:1467] Processing CreateTablet for tablet a95809cf5916449ba5eb0e120f33eadb (DEFAULT_TABLE table=test-table [id=d4eb6c29edd4496083fa226abd143d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:11.527926 6109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95809cf5916449ba5eb0e120f33eadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:11.530339 6189 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Bootstrap starting.
I20260929 03:23:11.530730 6191 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Bootstrap starting.
I20260929 03:23:11.530839 6190 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Bootstrap starting.
I20260929 03:23:11.531230 6189 tablet_bootstrap.cc:654] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:11.531418 6191 tablet_bootstrap.cc:654] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:11.531540 6190 tablet_bootstrap.cc:654] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:11.531844 6189 log.cc:824] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:11.531961 6191 log.cc:824] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:11.532128 6190 log.cc:824] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:11.532707 6191 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: No bootstrap required, opened a new log
I20260929 03:23:11.532737 6189 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: No bootstrap required, opened a new log
I20260929 03:23:11.532796 6189 ts_tablet_manager.cc:1397] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:23:11.532797 6191 ts_tablet_manager.cc:1397] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:23:11.532855 6190 tablet_bootstrap.cc:492] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: No bootstrap required, opened a new log
I20260929 03:23:11.532922 6190 ts_tablet_manager.cc:1397] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:23:11.534283 6191 raft_consensus.cc:348] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.534446 6191 raft_consensus.cc:374] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:11.534490 6191 raft_consensus.cc:729] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5df2d49b9248dd94dbb74c0b1e7af0, State: Initialized, Role: FOLLOWER
I20260929 03:23:11.534634 6191 consensus_queue.cc:260] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.534920 6191 ts_tablet_manager.cc:1428] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:23:11.534842 6189 raft_consensus.cc:348] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.534842 6190 raft_consensus.cc:348] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.534993 6189 raft_consensus.cc:374] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:11.535001 6190 raft_consensus.cc:374] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:11.535041 6190 raft_consensus.cc:729] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202b7d2d56434095b11de7e3f82d4ae2, State: Initialized, Role: FOLLOWER
I20260929 03:23:11.535041 6189 raft_consensus.cc:729] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494de34d612a47d3b17059a8914c317e, State: Initialized, Role: FOLLOWER
I20260929 03:23:11.535068 6175 heartbeater.cc:503] Master 127.27.68.126:37701 was elected leader, sending a full tablet report...
I20260929 03:23:11.535187 6190 consensus_queue.cc:260] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.535166 6189 consensus_queue.cc:260] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.535476 6189 ts_tablet_manager.cc:1428] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:23:11.535476 6190 ts_tablet_manager.cc:1428] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:23:11.535597 6041 heartbeater.cc:503] Master 127.27.68.126:37701 was elected leader, sending a full tablet report...
I20260929 03:23:11.535602 5907 heartbeater.cc:503] Master 127.27.68.126:37701 was elected leader, sending a full tablet report...
W20260929 03:23:11.587419 6042 tablet.cc:2369] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:23:11.686611 5908 tablet.cc:2369] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:23:11.727382 6195 raft_consensus.cc:482] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:23:11.727509 6195 raft_consensus.cc:504] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.727862 6195 leader_election.cc:288] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 202b7d2d56434095b11de7e3f82d4ae2 (127.27.68.65:44405), 494de34d612a47d3b17059a8914c317e (127.27.68.66:43669)
I20260929 03:23:11.730988 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.728178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45277 (local address 127.27.68.66:43669)
0929 03:23:11.728230 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.728233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.728502 (+ 269us) server_negotiation.cc:407] Connection header received
0929 03:23:11.728532 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.728533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.728546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.728649 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:11.728830 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.729301 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730259 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.730639 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.730738 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.730891 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.730924 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:23:11.730984 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.728201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:34051 (local address 127.27.68.65:44405)
0929 03:23:11.728248 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.728250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.728261 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:11.728329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.728330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.728344 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.728409 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:11.728692 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.729223 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730249 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.730614 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730673 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.730682 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.730855 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.730903 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:23:11.730988 6177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.728112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44405 (local address 127.27.68.67:34051)
0929 03:23:11.728182 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.728197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.728246 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.728525 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.728527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.728528 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:11.728608 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.728610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.729303 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.729304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.730151 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.730153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.730693 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.730778 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.730781 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.730832 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.730894 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":24}
I20260929 03:23:11.731007 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.728112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:43669 (local address 127.27.68.67:45277)
0929 03:23:11.728326 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.728337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:11.728405 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:11.728671 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:11.728680 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:11.728681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:11.728751 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.728753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.729377 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.729378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.730152 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:11.730154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.730744 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.730745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:11.730797 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.730800 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:11.730875 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:23:11.730911 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":22,"thread_start_us":98,"threads_started":1}
I20260929 03:23:11.731422 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95809cf5916449ba5eb0e120f33eadb" candidate_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202b7d2d56434095b11de7e3f82d4ae2" is_pre_election: true
I20260929 03:23:11.731609 5861 raft_consensus.cc:2454] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5df2d49b9248dd94dbb74c0b1e7af0 in term 0.
I20260929 03:23:11.731703 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95809cf5916449ba5eb0e120f33eadb" candidate_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494de34d612a47d3b17059a8914c317e" is_pre_election: true
I20260929 03:23:11.731881 5995 raft_consensus.cc:2454] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5df2d49b9248dd94dbb74c0b1e7af0 in term 0.
I20260929 03:23:11.731863 6062 leader_election.cc:302] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202b7d2d56434095b11de7e3f82d4ae2, 4f5df2d49b9248dd94dbb74c0b1e7af0; no voters:
I20260929 03:23:11.732035 6195 raft_consensus.cc:2781] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:23:11.732101 6195 raft_consensus.cc:482] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:23:11.732136 6195 raft_consensus.cc:3037] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:11.732892 6195 raft_consensus.cc:504] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.733047 6195 leader_election.cc:288] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [CANDIDATE]: Term 1 election: Requested vote from peers 202b7d2d56434095b11de7e3f82d4ae2 (127.27.68.65:44405), 494de34d612a47d3b17059a8914c317e (127.27.68.66:43669)
I20260929 03:23:11.733222 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95809cf5916449ba5eb0e120f33eadb" candidate_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494de34d612a47d3b17059a8914c317e"
I20260929 03:23:11.733239 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95809cf5916449ba5eb0e120f33eadb" candidate_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202b7d2d56434095b11de7e3f82d4ae2"
I20260929 03:23:11.733309 5995 raft_consensus.cc:3037] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:11.733310 5861 raft_consensus.cc:3037] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:11.734237 5861 raft_consensus.cc:2454] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5df2d49b9248dd94dbb74c0b1e7af0 in term 1.
I20260929 03:23:11.734237 5995 raft_consensus.cc:2454] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5df2d49b9248dd94dbb74c0b1e7af0 in term 1.
I20260929 03:23:11.734423 6063 leader_election.cc:302] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494de34d612a47d3b17059a8914c317e, 4f5df2d49b9248dd94dbb74c0b1e7af0; no voters:
I20260929 03:23:11.734575 6195 raft_consensus.cc:2781] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:11.734741 6195 raft_consensus.cc:686] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5df2d49b9248dd94dbb74c0b1e7af0, State: Running, Role: LEADER
I20260929 03:23:11.734858 6195 consensus_queue.cc:237] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } }
I20260929 03:23:11.735801 5715 catalog_manager.cc:5495] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5df2d49b9248dd94dbb74c0b1e7af0 (127.27.68.67). New cstate: current_term: 1 leader_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5df2d49b9248dd94dbb74c0b1e7af0" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 46229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 } health_report { overall_health: UNKNOWN } } }
W20260929 03:23:11.737005 6176 tablet.cc:2369] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:23:11.756019 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.753408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.27.68.67:46229)
0929 03:23:11.753474 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:11.753475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:11.753484 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:11.753569 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:11.753570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:11.753588 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:11.753676 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:23:11.753809 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.754270 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.755136 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:11.755390 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:11.755468 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:11.755636 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:23:11.755758 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:23:11.755857 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:11.755876 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 03:23:11.755897 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:23:11.757834 5995 raft_consensus.cc:1261] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 1 FOLLOWER]: Refusing update from remote peer 4f5df2d49b9248dd94dbb74c0b1e7af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:23:11.757838 5861 raft_consensus.cc:1261] T a95809cf5916449ba5eb0e120f33eadb P 202b7d2d56434095b11de7e3f82d4ae2 [term 1 FOLLOWER]: Refusing update from remote peer 4f5df2d49b9248dd94dbb74c0b1e7af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:23:11.758248 6199 consensus_queue.cc:1035] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "494de34d612a47d3b17059a8914c317e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 43669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:23:11.758395 6195 consensus_queue.cc:1035] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202b7d2d56434095b11de7e3f82d4ae2" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:23:11.761487 6204 mvcc.cc:205] Tried to move back new op lower bound from 7334511377437278208 to 7334511377346682880. Current Snapshot: MvccSnapshot[applied={T|T < 7334511377437278208}]
I20260929 03:23:11.763576 5715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:23:11.763688 5715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:23:11.764503 5715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcfb2234b00d4bd39597e6cfee110478: Sending DeleteTablet for 3 replicas of tablet a95809cf5916449ba5eb0e120f33eadb
I20260929 03:23:11.764871 6109 tablet_service.cc:1514] Processing DeleteTablet for tablet a95809cf5916449ba5eb0e120f33eadb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:60666
I20260929 03:23:11.764873 5975 tablet_service.cc:1514] Processing DeleteTablet for tablet a95809cf5916449ba5eb0e120f33eadb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:44614
I20260929 03:23:11.764973 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet a95809cf5916449ba5eb0e120f33eadb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:43478
I20260929 03:23:11.765105 6212 tablet_replica.cc:330] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: stopping tablet replica
I20260929 03:23:11.765231 6212 raft_consensus.cc:2229] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:23:11.765234 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5778
I20260929 03:23:11.765362 6212 raft_consensus.cc:2258] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:23:11.765622 6213 tablet_replica.cc:330] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: stopping tablet replica
I20260929 03:23:11.765779 6213 raft_consensus.cc:2229] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:23:11.765905 6213 raft_consensus.cc:2258] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:23:11.766402 6213 ts_tablet_manager.cc:1905] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:23:11.767120 6212 ts_tablet_manager.cc:1905] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:23:11.768277 6213 ts_tablet_manager.cc:1918] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:23:11.768383 6213 log.cc:1196] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/a95809cf5916449ba5eb0e120f33eadb
I20260929 03:23:11.768719 6213 ts_tablet_manager.cc:1939] T a95809cf5916449ba5eb0e120f33eadb P 494de34d612a47d3b17059a8914c317e: Deleting consensus metadata
I20260929 03:23:11.769193 6212 ts_tablet_manager.cc:1918] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:23:11.769222 5701 catalog_manager.cc:4841] TS 494de34d612a47d3b17059a8914c317e (127.27.68.66:43669): tablet a95809cf5916449ba5eb0e120f33eadb (table test-table [id=d4eb6c29edd4496083fa226abd143d0c]) successfully deleted
I20260929 03:23:11.769271 6212 log.cc:1196] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/a95809cf5916449ba5eb0e120f33eadb
I20260929 03:23:11.769550 6212 ts_tablet_manager.cc:1939] T a95809cf5916449ba5eb0e120f33eadb P 4f5df2d49b9248dd94dbb74c0b1e7af0: Deleting consensus metadata
I20260929 03:23:11.770015 5700 catalog_manager.cc:4841] TS 4f5df2d49b9248dd94dbb74c0b1e7af0 (127.27.68.67:46229): tablet a95809cf5916449ba5eb0e120f33eadb (table test-table [id=d4eb6c29edd4496083fa226abd143d0c]) successfully deleted
W20260929 03:23:11.772993 5700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:44405 (error 108)
I20260929 03:23:11.773180 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5912
I20260929 03:23:11.775668 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:11.775494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44405 (local address 127.0.0.1:43482)
0929 03:23:11.775558 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:11.775621 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 03:23:11.775857 5700 catalog_manager.cc:4584] TS 202b7d2d56434095b11de7e3f82d4ae2 (127.27.68.65:44405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a95809cf5916449ba5eb0e120f33eadb: Network error: Client connection negotiation failed: client connection to 127.27.68.65:44405: connect: Connection refused (error 111)
I20260929 03:23:11.779064 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 6046
I20260929 03:23:11.784993 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 5684
2026-09-29T03:23:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2084 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[6218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[6218](info): set up 2 sockets
Sep 29 03:23:11 dist-test-slave-fzwk krb5kdc[6218](info): commencing operation
krb5kdc: starting...
W20260929 03:23:12.828578 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:23:12 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:23:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:23:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangeradmin_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangeradmin_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangerlookup_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangerlookup_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
I20260929 03:23:12.990172 27921 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko51MCN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/postgres -l logfile start
2026-09-29 03:23:15.472 UTC [6248] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:23:15.472 UTC [6248] LOG: listening on IPv4 address "127.27.68.84", port 60763
2026-09-29 03:23:15.475 UTC [6248] LOG: listening on Unix socket "/tmp/.s.PGSQL.60763"
2026-09-29 03:23:15.496 UTC [6251] LOG: database system was shut down at 2026-09-29 03:23:14 UTC
2026-09-29 03:23:15.500 UTC [6248] LOG: database system is ready to accept connections
I20260929 03:23:16.424551 27921 mini_postgres.cc:96] Postgres bound to 60763
2026-09-29 03:23:16.430 UTC [6259] FATAL: database "slave" does not exist
127.27.68.84:60763 - accepting connections
I20260929 03:23:16.431201 27921 mini_ranger.cc:161] Starting Ranger...
I20260929 03:23:16.447014 27921 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:23:16.915984 27921 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:23:16.916074 27921 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin
2026-09-29 03:23:17,216 [I] DB FLAVOR :POSTGRES
2026-09-29 03:23:17,217 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:23:17,217 [I] Checking connection..
2026-09-29 03:23:17,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:23:17,454 [I] Checking connection passed.
2026-09-29 03:23:17,455 [I] --------- Verifying version history table ---------
2026-09-29 03:23:17,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:23:17,711 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:23:17,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:23:17,947 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:23:17,947 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:23:17,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:23:18.187 UTC [6323] WARNING: there is no transaction in progress
2026-09-29 03:23:18,193 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:23:18,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:23:18,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:23:18,444 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:23:18,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:23:18,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:23:18,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fzwk.c.gcp-upstream.internal','N') ;"
2026-09-29 03:23:19,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:23:19,415 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:23:19,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:23:19,671 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:23:19,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:23:19,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:23:20,159 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:23:20,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:23:21.118 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.134 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.150 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.166 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.185 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.207 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.272 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.278 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.293 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.304 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.315 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.329 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.339 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.346 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.353 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21.360 UTC [6431] WARNING: there is no transaction in progress
2026-09-29 03:23:21,992 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:23:21,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:23:22,240 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:23:22,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:23:22,517 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:23:22,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:23:22,789 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:23:22,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:23:23,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:23:23,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fzwk.c.gcp-upstream.internal';"
2026-09-29 03:23:23,544 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:23:23,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:23:23,815 [I] DB_PATCHES have already been applied
I20260929 03:23:23.821540 27921 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko51MCN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:23:23.821607 27921 mini_ranger.cc:193] Using host: 127.27.68.84
I20260929 03:23:23.824633 27921 mini_ranger.cc:227] Ranger admin URL: http://127.27.68.84:59327
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:23:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:23:24.931452 27921 mini_ranger.cc:160] Time spent starting Ranger: real 8.500s user 0.000s sys 0.009s
Sep 29, 2026 3:23:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:23:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 3:23:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:23:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:23:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:23:45.847023 27921 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangerkms_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/rangerkms_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.68.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.68.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:23:45 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652225, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:23:45.942766 27921 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:23:45.953192 27921 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:23:46.846939 27921 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:23:46.847020 27921 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms
I20260929 03:23:46.847038 27921 mini_ranger_kms.cc:227] Using postgres at 127.27.68.84:60763
2026-09-29 03:23:47,068 [I] DB FLAVOR :POSTGRES
2026-09-29 03:23:47,068 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:23:47,068 [I] Checking connection
2026-09-29 03:23:47,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:23:47,307 [I] connection success
2026-09-29 03:23:47,307 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:23:47,307 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:23:47,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:23:47,565 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:23:47,565 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:23:47,565 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:23:47,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:60763/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:23:47,827 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:23:48.321650 27921 mini_ranger_kms.cc:310] Created kms service
I20260929 03:23:48.577697 27921 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:23:48.624953 27921 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:23:48 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652225, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:49.018147 27921 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:23:49.018275 27921 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:23:49.018298 27921 mini_ranger_kms.cc:240] Using host: 127.27.68.84
I20260929 03:23:49.022024 27921 mini_ranger_kms.cc:276] Ranger KMS PID: 6611
I20260929 03:23:49.022173 27921 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.68.84:57795
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:23:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:23:51.309916 27921 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.367s user 0.000s sys 0.203s
I20260929 03:23:51.310022 27921 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:23:51.310105 27921 mini_ranger_kms.cc:401] 127.27.68.84:57795/kms/v1/keys
Sep 29 03:23:51 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:23:52 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
Sep 29 03:23:52 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652231, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
Sep 29 03:23:54 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:23:54.695108 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42159
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42159
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:57795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:54.780090 6662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:54.780298 6662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:54.780321 6662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:54.781454 6662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:23:54.781493 6662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:54.781507 6662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:54.781522 6662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:23:54.781536 6662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:23:54.783064 6662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.68.84:57795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42159
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:54.783344 6662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:54.783540 6662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:54.789242 6670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:54.789211 6667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:54.789211 6668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:54.789247 6662 server_base.cc:1034] running on GCE node
I20260929 03:23:54.789721 6662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:54.790091 6662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:54.791287 6662 hybrid_clock.cc:648] HybridClock initialized: now 1790652234791275 us; error 53 us; skew 500 ppm
Sep 29 03:23:54 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652234, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:54.793915 6662 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:23:54.794377 6662 webserver.cc:460] Webserver started at http://127.27.68.126:34407/ using document root <none> and password file <none>
I20260929 03:23:54.794625 6662 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:54.794685 6662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:54.794834 6662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:23:54 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652234, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:54.848943 6662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "243fcf84bf424f10aeaa263c70af20d1"
format_stamp: "Formatted at 2026-09-29 03:23:54 on dist-test-slave-fzwk"
server_key: "9e66d95e2eaef1709cb0f5ff5c4c8895"
server_key_iv: "fbf70b2b49c2e178a8c50909ae931cd3"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:54.849375 6662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "243fcf84bf424f10aeaa263c70af20d1"
format_stamp: "Formatted at 2026-09-29 03:23:54 on dist-test-slave-fzwk"
server_key: "9e66d95e2eaef1709cb0f5ff5c4c8895"
server_key_iv: "fbf70b2b49c2e178a8c50909ae931cd3"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:54.850862 6662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 03:23:54.863173 6677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:54.863456 6662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:23:54.863572 6662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "243fcf84bf424f10aeaa263c70af20d1"
format_stamp: "Formatted at 2026-09-29 03:23:54 on dist-test-slave-fzwk"
server_key: "9e66d95e2eaef1709cb0f5ff5c4c8895"
server_key_iv: "fbf70b2b49c2e178a8c50909ae931cd3"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:54.863690 6662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:54.883653 6662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:54.885829 6662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:54.886003 6662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:54.890825 6662 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42159
I20260929 03:23:54.890841 6729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42159 every 8 connection(s)
I20260929 03:23:54.891393 6662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:23:54.892150 6730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:54.894484 6730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: Bootstrap starting.
I20260929 03:23:54.895112 6730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:54.895447 6730 log.cc:824] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:54.896162 6730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: No bootstrap required, opened a new log
I20260929 03:23:54.898195 6730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } }
I20260929 03:23:54.898344 6730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:54.898366 6730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243fcf84bf424f10aeaa263c70af20d1, State: Initialized, Role: FOLLOWER
I20260929 03:23:54.898523 6730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } }
I20260929 03:23:54.898605 6730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:23:54.898628 6730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:23:54.898696 6730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:54.899416 6730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } }
I20260929 03:23:54.899506 6730 leader_election.cc:302] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243fcf84bf424f10aeaa263c70af20d1; no voters:
I20260929 03:23:54.899745 6730 leader_election.cc:288] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:23:54.899771 6733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:54.899910 6733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [term 1 LEADER]: Becoming Leader. State: Replica: 243fcf84bf424f10aeaa263c70af20d1, State: Running, Role: LEADER
I20260929 03:23:54.900040 6730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:23:54.900056 6733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } }
I20260929 03:23:54.900501 6734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "243fcf84bf424f10aeaa263c70af20d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } } }
I20260929 03:23:54.900601 6734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:54.900838 6735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243fcf84bf424f10aeaa263c70af20d1. Latest consensus state: current_term: 1 leader_uuid: "243fcf84bf424f10aeaa263c70af20d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243fcf84bf424f10aeaa263c70af20d1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42159 } } }
I20260929 03:23:54.901101 6735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:23:54.901260 6740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:23:54.901413 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 6662
I20260929 03:23:54.901520 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:23:54.901633 6730 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:23:54.901726 27921 external_mini_cluster.cc:1428] Setting key b44cf3740484db5ab69adfd57666a2bf
I20260929 03:23:54.901813 6740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:23:54.903657 6740 catalog_manager.cc:1347] Generated new cluster ID: f596a9cce5b84c34b2fd6a896cd1645f
I20260929 03:23:54.903716 6740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:23:54 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:54.928874 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:54.902461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.27.68.126:42159)
0929 03:23:54.902861 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:54.902869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:54.902893 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:23:54.903037 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:54.903044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:54.903107 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:54.903316 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:54.904193 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:54.905640 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:54.906694 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:54.907353 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:54.907441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:54.912270 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:54.912312 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:54.912312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:54.912400 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:54.915746 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:54.927179 (+ 11433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:54.927185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:54.927185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:54.927374 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:54.927603 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:54.927604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:54.927605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:54.928001 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:54.928156 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:54.928280 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:23:54.928467 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":78,"threads_started":1}
I20260929 03:23:54.933010 6740 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:23:54.933863 6740 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:23:54.941888 6740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: Generated new TSK 0
I20260929 03:23:54.942335 6740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:23:55.590273 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:55.683943 6775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:55.684182 6775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:55.684227 6775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:55.685556 6775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:55.685631 6775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:55.685678 6775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:23:55.687417 6775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.68.84:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:55.687696 6775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:55.687963 6775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:55.693086 6781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:55.693131 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:55.693190 6783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:55.693545 6775 server_base.cc:1034] running on GCE node
I20260929 03:23:55.693781 6775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:55.694056 6775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:55.695273 6775 hybrid_clock.cc:648] HybridClock initialized: now 1790652235695259 us; error 34 us; skew 500 ppm
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:55.698005 6775 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:23:55.698474 6775 webserver.cc:460] Webserver started at http://127.27.68.65:40571/ using document root <none> and password file <none>
I20260929 03:23:55.698729 6775 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:55.698851 6775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:55.699009 6775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:55.713785 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "8a97a9f7c3ee471db5058d0645406f72"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "5b5463e979edcb0f8083527682778288"
server_key_iv: "032038ffd73fb971ba7c7619490f8051"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.714273 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "8a97a9f7c3ee471db5058d0645406f72"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "5b5463e979edcb0f8083527682778288"
server_key_iv: "032038ffd73fb971ba7c7619490f8051"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.716341 6775 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 03:23:55.730955 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.731513 6775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:23:55.731604 6775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "8a97a9f7c3ee471db5058d0645406f72"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "5b5463e979edcb0f8083527682778288"
server_key_iv: "032038ffd73fb971ba7c7619490f8051"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.731704 6775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:55.751569 6775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:55.754300 6775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:55.754567 6775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:55.754914 6775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:55.755376 6775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:55.755488 6775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.755599 6775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:55.755679 6775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.762696 6775 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:38841
I20260929 03:23:55.762822 6903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:38841 every 8 connection(s)
I20260929 03:23:55.763310 6775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:23:55.768697 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 6775
I20260929 03:23:55.768811 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:23:55.768999 27921 external_mini_cluster.cc:1428] Setting key 717e49c353c7e125aaa9785ca85da8a2
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:55.775604 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:55.764106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42159 (local address 127.27.68.65:42383)
0929 03:23:55.764309 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:55.764378 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:55.765053 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:55.765527 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:55.765532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:55.765787 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:55.766081 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:55.766089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:55.767210 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:55.767211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:55.768557 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:55.768560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:55.769132 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:55.769133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:55.769283 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:55.769924 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:55.769935 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:55.771709 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:55.773838 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:55.773846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:55.773849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:55.774163 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:55.774408 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:55.774409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:55.774409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:55.774549 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:55.774847 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:55.774853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:55.775039 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 03:23:55.775151 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:23:55.775341 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260929 03:23:55.776079 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:55.764145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:42383 (local address 127.27.68.126:42159)
0929 03:23:55.765275 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:55.765278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:55.765287 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:23:55.765314 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:55.765314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:55.765328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:55.765401 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:55.766185 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:55.767111 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:55.768666 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:55.768995 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:55.769060 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:55.771827 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:55.771829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:55.771829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:55.771837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:55.773741 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:55.774267 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:55.774269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:55.774269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:55.774340 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:55.774618 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:55.774619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:55.774619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:55.774764 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:55.774822 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:55.775873 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 03:23:55.775978 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":84,"threads_started":1}
I20260929 03:23:55.776710 6904 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42159
I20260929 03:23:55.776814 6904 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:55.777026 6904 heartbeater.cc:511] Master 127.27.68.126:42159 requested a full tablet report, sending...
I20260929 03:23:55.777637 6694 ts_manager.cc:194] Registered new tserver with Master: 8a97a9f7c3ee471db5058d0645406f72 (127.27.68.65:38841)
I20260929 03:23:55.778555 6694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:42383
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:23:55.802654 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:55.916247 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:55.916556 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:55.916599 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:55.918385 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:55.918453 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:55.918495 6912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:23:55.920831 6912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.68.84:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:55.921123 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:55.921438 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:55.927935 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:55.928005 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:55.928398 6919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:55.928537 6912 server_base.cc:1034] running on GCE node
I20260929 03:23:55.928755 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:55.929044 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:55.930229 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790652235930197 us; error 46 us; skew 500 ppm
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:55.933795 6912 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:23:55.934415 6912 webserver.cc:460] Webserver started at http://127.27.68.66:43561/ using document root <none> and password file <none>
I20260929 03:23:55.934703 6912 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:55.934780 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:55.934984 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:55.950093 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "91ed051ddc684a1e9b6fe07205fdbd75"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "3a4b36b87418681d2510e3376d7966ad"
server_key_iv: "6241c7267fd89a74b61082c8c035e8bf"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.950739 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "91ed051ddc684a1e9b6fe07205fdbd75"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "3a4b36b87418681d2510e3376d7966ad"
server_key_iv: "6241c7267fd89a74b61082c8c035e8bf"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.956698 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260929 03:23:55.972102 6929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.972453 6912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:23:55.972621 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "91ed051ddc684a1e9b6fe07205fdbd75"
format_stamp: "Formatted at 2026-09-29 03:23:55 on dist-test-slave-fzwk"
server_key: "3a4b36b87418681d2510e3376d7966ad"
server_key_iv: "6241c7267fd89a74b61082c8c035e8bf"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:55.972831 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 03:23:55 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:55.983716 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:55.986392 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:55.986637 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:55.986975 6912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:55.987468 6912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:55.987560 6912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.987634 6912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:55.987690 6912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:55.994025 6912 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:44191
I20260929 03:23:55.994558 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:23:55.999094 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 6912
I20260929 03:23:55.999241 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:23:55.999451 27921 external_mini_cluster.cc:1428] Setting key 10611c925e3242370f3ac91d47534c87
I20260929 03:23:56.007941 7042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:44191 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Sep 29 03:23:56 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
I20260929 03:23:56.036326 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.009093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:40937 (local address 127.27.68.126:42159)
0929 03:23:56.009148 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.009151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.011168 (+ 2017us) server_negotiation.cc:407] Connection header received
0929 03:23:56.012016 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.012017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.012034 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.012080 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:56.012939 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.013690 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.018625 (+ 4935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.018962 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.019014 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.022272 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:56.022274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:56.022275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:56.022283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:56.024134 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.032119 (+ 7985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.032122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.032122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.032231 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.035419 (+ 3188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.035421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.035422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.035606 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:56.035666 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.036080 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.036202 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:23:56.036871 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.008739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42159 (local address 127.27.68.66:40937)
0929 03:23:56.010989 (+ 2250us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.011063 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.011897 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.012203 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.012208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.012504 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:56.012836 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.012844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.017091 (+ 4247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.017093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.018489 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.018493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.019121 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.019122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.019303 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.019987 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:56.020000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:56.021722 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:56.024238 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:56.024246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:56.024249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:56.031965 (+ 7716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:56.032319 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:56.032320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:56.032320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:56.035284 (+ 2964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:56.035702 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:56.035709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:56.035907 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.036431 (+ 524us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.036620 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2175,"thread_start_us":115,"threads_started":1}
I20260929 03:23:56.037719 7043 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42159
I20260929 03:23:56.037842 7043 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:56.038057 7043 heartbeater.cc:511] Master 127.27.68.126:42159 requested a full tablet report, sending...
I20260929 03:23:56.038480 6694 ts_manager.cc:194] Registered new tserver with Master: 91ed051ddc684a1e9b6fe07205fdbd75 (127.27.68.66:44191)
I20260929 03:23:56.039027 6694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:40937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:23:56.053737 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:23:56.185559 7048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:23:56.185961 7048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:23:56.186064 7048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:23:56.188037 7048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:23:56.188174 7048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:23:56.188272 7048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:23:56.190776 7048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.68.84:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:23:56.191431 7048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:23:56.191805 7048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:23:56.199054 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:56.200567 7048 server_base.cc:1034] running on GCE node
W20260929 03:23:56.200881 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:23:56.205469 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:23:56.205951 7048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:23:56.206307 7048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:23:56.213747 7048 hybrid_clock.cc:648] HybridClock initialized: now 1790652236213684 us; error 64 us; skew 500 ppm
Sep 29 03:23:56 dist-test-slave-fzwk krb5kdc[6218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:23:56.217535 7048 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:23:56.218629 7048 webserver.cc:460] Webserver started at http://127.27.68.67:39013/ using document root <none> and password file <none>
I20260929 03:23:56.219005 7048 fs_manager.cc:359] Metadata directory not provided
I20260929 03:23:56.219161 7048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:23:56.219432 7048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:23:56 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:56.245323 7048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "ab105a393d47476aa27bba6ee9f3b972"
format_stamp: "Formatted at 2026-09-29 03:23:56 on dist-test-slave-fzwk"
server_key: "55c833f48dd84bb2cd95ba5867c586ce"
server_key_iv: "4be63c74ab9f3b457e38df3d83e38ccc"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:56.245951 7048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "ab105a393d47476aa27bba6ee9f3b972"
format_stamp: "Formatted at 2026-09-29 03:23:56 on dist-test-slave-fzwk"
server_key: "55c833f48dd84bb2cd95ba5867c586ce"
server_key_iv: "4be63c74ab9f3b457e38df3d83e38ccc"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:56.252563 7048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260929 03:23:56.270107 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:56.270387 7048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:23:56.271405 7048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "ab105a393d47476aa27bba6ee9f3b972"
format_stamp: "Formatted at 2026-09-29 03:23:56 on dist-test-slave-fzwk"
server_key: "55c833f48dd84bb2cd95ba5867c586ce"
server_key_iv: "4be63c74ab9f3b457e38df3d83e38ccc"
server_key_version: "kuduclusterkey@0"
I20260929 03:23:56.271533 7048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:23:56.300776 7048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:23:56.310652 7048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:23:56.310833 7048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:23:56.311120 7048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:23:56.311564 7048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:23:56.311620 7048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:56.311661 7048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:23:56.311683 7048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:23:56.318018 7048 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:39577
I20260929 03:23:56.318481 7048 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:23:56.323835 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:39577 every 8 connection(s)
I20260929 03:23:56.325361 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 7048
I20260929 03:23:56.325477 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:23:56.325654 27921 external_mini_cluster.cc:1428] Setting key 7fe219dea7f26198e7bf90724deface4
Sep 29 03:23:56 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652236, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:23:56.340427 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.325073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:57615 (local address 127.27.68.126:42159)
0929 03:23:56.325126 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.325128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.327440 (+ 2312us) server_negotiation.cc:407] Connection header received
0929 03:23:56.328114 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.328115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.328128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.328157 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:56.329080 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.329879 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.331484 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.331775 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.331820 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.336812 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:56.336815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:56.336815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:56.336824 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:56.338644 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.339257 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.339259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.339259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.339326 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.339650 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.339651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.339651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.339806 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:56.339858 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.340237 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.340326 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:23:56.341377 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.325968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42159 (local address 127.27.68.67:57615)
0929 03:23:56.327270 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.327327 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.327994 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.328343 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.328347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.328620 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:23:56.328976 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.328983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.329995 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.329996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.331205 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.331207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.331913 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.331914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.332058 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.332729 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:23:56.332740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:23:56.336671 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:23:56.338762 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:56.338771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:56.338774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:56.339125 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:56.339398 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:23:56.339399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:23:56.339399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:23:56.339570 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:23:56.339890 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:23:56.339897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:23:56.340078 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.340932 (+ 854us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.341122 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"thread_start_us":101,"threads_started":1}
I20260929 03:23:56.342073 7178 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42159
I20260929 03:23:56.342206 7178 heartbeater.cc:464] Registering TS with master...
I20260929 03:23:56.342424 7178 heartbeater.cc:511] Master 127.27.68.126:42159 requested a full tablet report, sending...
I20260929 03:23:56.342800 6694 ts_manager.cc:194] Registered new tserver with Master: ab105a393d47476aa27bba6ee9f3b972 (127.27.68.67:39577)
I20260929 03:23:56.343312 6694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:57615
I20260929 03:23:56.347936 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:23:56.359357 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.349532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.27.68.126:42159)
0929 03:23:56.349577 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.349579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.351485 (+ 1906us) server_negotiation.cc:407] Connection header received
0929 03:23:56.351597 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.351598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.351610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.351640 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:23:56.352466 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.353240 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.354377 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.354699 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.354742 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.356383 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:23:56.356385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:23:56.356385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:23:56.356393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:23:56.358061 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.358489 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.358491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.358491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.358550 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:23:56.358746 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:23:56.358747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:23:56.358747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:23:56.358873 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:23:56.358945 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.359153 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.359257 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:23:56.361496 6694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:23:56.363955 6694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:23:56.397747 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38841 (local address 127.0.0.1:41096)
0929 03:23:56.391731 (+ 6462us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.391741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.391784 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.392082 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.392085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.392087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:56.392190 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.392193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.393887 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.393887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.396008 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.396010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.397408 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.397408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.397509 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.397524 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.397600 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.397656 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6445,"thread_start_us":92,"threads_started":1}
I20260929 03:23:56.398226 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.27.68.65:38841)
0929 03:23:56.391366 (+ 6002us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.391371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.391843 (+ 472us) server_negotiation.cc:407] Connection header received
0929 03:23:56.391886 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.391889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.391906 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.391977 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:56.392277 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.393457 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.396085 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.396889 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.396963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.397371 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.398092 (+ 721us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.398146 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5957,"thread_start_us":72,"threads_started":1}
I20260929 03:23:56.401859 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 2794b2ca6dfc464482e318108ff65424 (DEFAULT_TABLE table=test-table [id=3c64d620fd284ab0885d6f9293ef564d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:56.402271 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2794b2ca6dfc464482e318108ff65424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:56.402606 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.27.68.66:44191)
0929 03:23:56.391347 (+ 5731us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.391353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.395887 (+ 4534us) server_negotiation.cc:407] Connection header received
0929 03:23:56.395947 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.395950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.395967 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.396062 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:56.396887 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.398022 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.401344 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.402117 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.402172 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.402246 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.402479 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.402523 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5675,"thread_start_us":90,"threads_started":1}
I20260929 03:23:56.402858 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:44191 (local address 127.0.0.1:58994)
0929 03:23:56.395811 (+ 10300us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.395822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.395871 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.396176 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.396178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.396180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:56.396224 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.396226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.398129 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.398129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.401155 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.401158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.402309 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.402310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.402370 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.402374 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.402741 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.402789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10283,"thread_start_us":79,"threads_started":1}
I20260929 03:23:56.407075 6966 tablet_service.cc:1467] Processing CreateTablet for tablet 2794b2ca6dfc464482e318108ff65424 (DEFAULT_TABLE table=test-table [id=3c64d620fd284ab0885d6f9293ef564d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:56.407500 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2794b2ca6dfc464482e318108ff65424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:56.412985 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:39577 (local address 127.0.0.1:37982)
0929 03:23:56.391464 (+ 6230us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.391528 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.391605 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.395659 (+ 4054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.395660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.395663 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:56.395712 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.395714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.399662 (+ 3948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.399670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.400785 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.400789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.412652 (+ 11863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.412654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.412733 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.412738 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.412829 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.412886 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6182,"thread_start_us":92,"threads_started":1}
I20260929 03:23:56.413532 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.385511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.27.68.67:39577)
0929 03:23:56.395387 (+ 9876us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.395394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.395405 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:23:56.395456 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.395458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.395476 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.395562 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:56.398428 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.399560 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.411544 (+ 11984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.412422 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.412525 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.412606 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.413384 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.413449 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9829,"thread_start_us":83,"threads_started":1}
I20260929 03:23:56.414345 7112 tablet_service.cc:1467] Processing CreateTablet for tablet 2794b2ca6dfc464482e318108ff65424 (DEFAULT_TABLE table=test-table [id=3c64d620fd284ab0885d6f9293ef564d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:23:56.414729 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2794b2ca6dfc464482e318108ff65424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:23:56.419375 7197 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Bootstrap starting.
I20260929 03:23:56.420521 7197 tablet_bootstrap.cc:654] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:56.419143 7196 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: Bootstrap starting.
I20260929 03:23:56.421399 7196 tablet_bootstrap.cc:654] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:56.421777 7196 log.cc:824] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:56.422152 7197 log.cc:824] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:56.422735 7198 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Bootstrap starting.
I20260929 03:23:56.423772 7198 tablet_bootstrap.cc:654] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Neither blocks nor log segments found. Creating new log.
I20260929 03:23:56.424104 7198 log.cc:824] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Log is configured to *not* fsync() on all Append() calls
I20260929 03:23:56.425403 7196 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: No bootstrap required, opened a new log
I20260929 03:23:56.425540 7196 ts_tablet_manager.cc:1397] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:23:56.428193 7196 raft_consensus.cc:348] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.428536 7196 raft_consensus.cc:374] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:56.428620 7196 raft_consensus.cc:729] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a97a9f7c3ee471db5058d0645406f72, State: Initialized, Role: FOLLOWER
I20260929 03:23:56.428822 7196 consensus_queue.cc:260] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.429178 7196 ts_tablet_manager.cc:1428] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 03:23:56.429371 6904 heartbeater.cc:503] Master 127.27.68.126:42159 was elected leader, sending a full tablet report...
I20260929 03:23:56.430831 7197 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: No bootstrap required, opened a new log
I20260929 03:23:56.430963 7197 ts_tablet_manager.cc:1397] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260929 03:23:56.433460 7197 raft_consensus.cc:348] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.433729 7197 raft_consensus.cc:374] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:56.433806 7197 raft_consensus.cc:729] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ed051ddc684a1e9b6fe07205fdbd75, State: Initialized, Role: FOLLOWER
I20260929 03:23:56.434007 7197 consensus_queue.cc:260] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.434369 7197 ts_tablet_manager.cc:1428] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:23:56.434540 7043 heartbeater.cc:503] Master 127.27.68.126:42159 was elected leader, sending a full tablet report...
I20260929 03:23:56.435791 7198 tablet_bootstrap.cc:492] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: No bootstrap required, opened a new log
I20260929 03:23:56.435920 7198 ts_tablet_manager.cc:1397] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260929 03:23:56.437990 7198 raft_consensus.cc:348] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.438222 7198 raft_consensus.cc:374] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:23:56.438292 7198 raft_consensus.cc:729] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab105a393d47476aa27bba6ee9f3b972, State: Initialized, Role: FOLLOWER
I20260929 03:23:56.438480 7198 consensus_queue.cc:260] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.438799 7198 ts_tablet_manager.cc:1428] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:23:56.438952 7178 heartbeater.cc:503] Master 127.27.68.126:42159 was elected leader, sending a full tablet report...
I20260929 03:23:56.447574 7202 raft_consensus.cc:482] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:23:56.447826 7202 raft_consensus.cc:504] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.448729 7202 leader_election.cc:288] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab105a393d47476aa27bba6ee9f3b972 (127.27.68.67:39577), 91ed051ddc684a1e9b6fe07205fdbd75 (127.27.68.66:44191)
I20260929 03:23:56.452584 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.448628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40137 (local address 127.27.68.66:44191)
0929 03:23:56.448744 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.448746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.448990 (+ 244us) server_negotiation.cc:407] Connection header received
0929 03:23:56.449101 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.449102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.449111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.449139 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:56.449422 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.450189 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.451648 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.452184 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.452240 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.452249 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.452466 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.452509 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 03:23:56.453074 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.448567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:44191 (local address 127.27.68.65:40137)
0929 03:23:56.448919 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.448931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.449029 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.449206 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.449206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.449209 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:56.449296 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.449298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.450334 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.450336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.451547 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.451551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.452307 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.452308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.452377 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.452380 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.452951 (+ 571us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.452996 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":107,"threads_started":1}
I20260929 03:23:56.453524 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2794b2ca6dfc464482e318108ff65424" candidate_uuid: "8a97a9f7c3ee471db5058d0645406f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" is_pre_election: true
I20260929 03:23:56.453687 6978 raft_consensus.cc:2454] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a97a9f7c3ee471db5058d0645406f72 in term 0.
I20260929 03:23:56.453981 6794 leader_election.cc:302] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a97a9f7c3ee471db5058d0645406f72, 91ed051ddc684a1e9b6fe07205fdbd75; no voters:
I20260929 03:23:56.454161 7202 raft_consensus.cc:2781] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:23:56.454205 7202 raft_consensus.cc:482] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:23:56.454236 7202 raft_consensus.cc:3037] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:56.455061 7202 raft_consensus.cc:504] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.455186 7202 leader_election.cc:288] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [CANDIDATE]: Term 1 election: Requested vote from peers ab105a393d47476aa27bba6ee9f3b972 (127.27.68.67:39577), 91ed051ddc684a1e9b6fe07205fdbd75 (127.27.68.66:44191)
I20260929 03:23:56.455541 6978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2794b2ca6dfc464482e318108ff65424" candidate_uuid: "8a97a9f7c3ee471db5058d0645406f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91ed051ddc684a1e9b6fe07205fdbd75"
I20260929 03:23:56.455614 6978 raft_consensus.cc:3037] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:56.456426 6978 raft_consensus.cc:2454] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a97a9f7c3ee471db5058d0645406f72 in term 1.
I20260929 03:23:56.456974 6794 leader_election.cc:302] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a97a9f7c3ee471db5058d0645406f72, 91ed051ddc684a1e9b6fe07205fdbd75; no voters:
I20260929 03:23:56.457082 7202 raft_consensus.cc:2781] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:23:56.457231 7202 raft_consensus.cc:686] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [term 1 LEADER]: Becoming Leader. State: Replica: 8a97a9f7c3ee471db5058d0645406f72, State: Running, Role: LEADER
I20260929 03:23:56.457311 7202 consensus_queue.cc:237] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } }
I20260929 03:23:56.458160 6694 catalog_manager.cc:5495] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a97a9f7c3ee471db5058d0645406f72 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "8a97a9f7c3ee471db5058d0645406f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a97a9f7c3ee471db5058d0645406f72" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 } health_report { overall_health: UNKNOWN } } }
I20260929 03:23:56.462321 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.448642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:60819 (local address 127.27.68.67:39577)
0929 03:23:56.448692 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.448694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.459274 (+ 10580us) server_negotiation.cc:407] Connection header received
0929 03:23:56.459379 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.459380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.459394 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.459421 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:23:56.459591 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.460282 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.461502 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.461988 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.462032 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.462041 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.462220 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.462256 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:23:56.462685 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.448575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:39577 (local address 127.27.68.65:60819)
0929 03:23:56.459183 (+ 10608us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.459195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:23:56.459306 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:23:56.459491 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:23:56.459491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:23:56.459493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:23:56.459529 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.459530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.460394 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.460395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.461394 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:23:56.461396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.462096 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.462097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:23:56.462151 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.462154 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:23:56.462582 (+ 428us) client_negotiation.cc:240] Negotiation successful
0929 03:23:56.462627 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10593,"thread_start_us":106,"threads_started":1}
I20260929 03:23:56.463207 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2794b2ca6dfc464482e318108ff65424" candidate_uuid: "8a97a9f7c3ee471db5058d0645406f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab105a393d47476aa27bba6ee9f3b972" is_pre_election: true
I20260929 03:23:56.463392 7127 raft_consensus.cc:2454] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a97a9f7c3ee471db5058d0645406f72 in term 0.
I20260929 03:23:56.463500 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2794b2ca6dfc464482e318108ff65424" candidate_uuid: "8a97a9f7c3ee471db5058d0645406f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab105a393d47476aa27bba6ee9f3b972"
I20260929 03:23:56.463543 7126 raft_consensus.cc:3037] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:23:56.464367 7126 raft_consensus.cc:2454] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a97a9f7c3ee471db5058d0645406f72 in term 1.
I20260929 03:23:56.483448 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.480060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.27.68.65:38841)
0929 03:23:56.480103 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:23:56.480105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:23:56.480113 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:23:56.480157 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:23:56.480157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:23:56.480181 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:23:56.480248 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:23:56.480460 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.481208 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.482310 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:23:56.482591 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:23:56.482652 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:23:56.482847 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:23:56.482988 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:23:56.483060 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:23:56.483328 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 03:23:56.483351 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:23:56.494336 6978 raft_consensus.cc:1261] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 1 FOLLOWER]: Refusing update from remote peer 8a97a9f7c3ee471db5058d0645406f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:23:56.494722 7126 raft_consensus.cc:1261] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 1 FOLLOWER]: Refusing update from remote peer 8a97a9f7c3ee471db5058d0645406f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:23:56.495349 7202 consensus_queue.cc:1035] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab105a393d47476aa27bba6ee9f3b972" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:23:56.495563 7202 consensus_queue.cc:1035] T 2794b2ca6dfc464482e318108ff65424 P 8a97a9f7c3ee471db5058d0645406f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91ed051ddc684a1e9b6fe07205fdbd75" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:23:56.502389 7210 mvcc.cc:205] Tried to move back new op lower bound from 7334511560676929536 to 7334511560530149376. Current Snapshot: MvccSnapshot[applied={T|T < 7334511560676929536}]
I20260929 03:23:56.502609 7213 mvcc.cc:205] Tried to move back new op lower bound from 7334511560676929536 to 7334511560530149376. Current Snapshot: MvccSnapshot[applied={T|T < 7334511560676929536}]
I20260929 03:23:56.505300 7211 mvcc.cc:205] Tried to move back new op lower bound from 7334511560676929536 to 7334511560530149376. Current Snapshot: MvccSnapshot[applied={T|T < 7334511560676929536}]
I20260929 03:23:56.508709 6694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:23:56.508838 6694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:23:56.510218 6694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 243fcf84bf424f10aeaa263c70af20d1: Sending DeleteTablet for 3 replicas of tablet 2794b2ca6dfc464482e318108ff65424
I20260929 03:23:56.510713 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet 2794b2ca6dfc464482e318108ff65424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:56 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:41096
I20260929 03:23:56.510955 6966 tablet_service.cc:1514] Processing DeleteTablet for tablet 2794b2ca6dfc464482e318108ff65424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:56 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:58994
I20260929 03:23:56.511514 7112 tablet_service.cc:1514] Processing DeleteTablet for tablet 2794b2ca6dfc464482e318108ff65424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:23:56 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:37982
I20260929 03:23:56.512184 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 6775
I20260929 03:23:56.519562 7222 tablet_replica.cc:330] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: stopping tablet replica
I20260929 03:23:56.519743 7222 raft_consensus.cc:2229] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:23:56.519894 7222 raft_consensus.cc:2258] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 03:23:56 dist-test-slave-fzwk krb5kdc[6218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652235, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:23:56.522225 7222 ts_tablet_manager.cc:1905] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:23:56.525168 7221 tablet_replica.cc:330] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: stopping tablet replica
I20260929 03:23:56.525323 7221 raft_consensus.cc:2229] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:23:56.525453 7221 raft_consensus.cc:2258] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:23:56.525906 7222 ts_tablet_manager.cc:1918] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:23:56.526007 7222 log.cc:1196] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/2794b2ca6dfc464482e318108ff65424
I20260929 03:23:56.526005 7221 ts_tablet_manager.cc:1905] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:23:56.526593 7222 ts_tablet_manager.cc:1939] T 2794b2ca6dfc464482e318108ff65424 P ab105a393d47476aa27bba6ee9f3b972: Deleting consensus metadata
I20260929 03:23:56.527136 6680 catalog_manager.cc:4841] TS ab105a393d47476aa27bba6ee9f3b972 (127.27.68.67:39577): tablet 2794b2ca6dfc464482e318108ff65424 (table test-table [id=3c64d620fd284ab0885d6f9293ef564d]) successfully deleted
I20260929 03:23:56.527613 7221 ts_tablet_manager.cc:1918] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:23:56.527693 7221 log.cc:1196] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/2794b2ca6dfc464482e318108ff65424
I20260929 03:23:56.527947 7221 ts_tablet_manager.cc:1939] T 2794b2ca6dfc464482e318108ff65424 P 91ed051ddc684a1e9b6fe07205fdbd75: Deleting consensus metadata
I20260929 03:23:56.528427 6681 catalog_manager.cc:4841] TS 91ed051ddc684a1e9b6fe07205fdbd75 (127.27.68.66:44191): tablet 2794b2ca6dfc464482e318108ff65424 (table test-table [id=3c64d620fd284ab0885d6f9293ef564d]) successfully deleted
W20260929 03:23:56.551412 6679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:38841 (error 108)
I20260929 03:23:56.552671 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:23:56.552529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38841 (local address 127.0.0.1:41108)
0929 03:23:56.552587 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:23:56.552623 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:23:56.552747 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 6912
W20260929 03:23:56.552976 6679 catalog_manager.cc:4584] TS 8a97a9f7c3ee471db5058d0645406f72 (127.27.68.65:38841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2794b2ca6dfc464482e318108ff65424: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38841: connect: Connection refused (error 111)
I20260929 03:23:56.558482 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 7048
I20260929 03:23:56.568885 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 6662
I20260929 03:23:56.575294 27921 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:23:56.930316 27921 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:23:56.930428 27921 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:23:57.275403 27921 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:23:57.275 UTC [6248] LOG: received smart shutdown request
2026-09-29 03:23:57.279 UTC [6248] LOG: background worker "logical replication launcher" (PID 6257) exited with exit code 1
2026-09-29 03:23:57.279 UTC [6252] LOG: shutting down
2026-09-29 03:23:57.435 UTC [6248] LOG: database system is shut down
2026-09-29T03:23:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45760 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:23:57 dist-test-slave-fzwk krb5kdc[7232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:23:57 dist-test-slave-fzwk krb5kdc[7232](info): set up 2 sockets
Sep 29 03:23:57 dist-test-slave-fzwk krb5kdc[7232](info): commencing operation
krb5kdc: starting...
W20260929 03:23:58.591548 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:23:58 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:23:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangeradmin_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangeradmin_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangerlookup_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangerlookup_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
I20260929 03:23:58.755759 27921 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko51MCN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/postgres -l logfile start
2026-09-29 03:24:01.249 UTC [7262] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:24:01.249 UTC [7262] LOG: listening on IPv4 address "127.27.68.84", port 44019
2026-09-29 03:24:01.253 UTC [7262] LOG: listening on Unix socket "/tmp/.s.PGSQL.44019"
2026-09-29 03:24:01.288 UTC [7265] LOG: database system was shut down at 2026-09-29 03:23:59 UTC
2026-09-29 03:24:01.293 UTC [7262] LOG: database system is ready to accept connections
I20260929 03:24:02.247664 27921 mini_postgres.cc:96] Postgres bound to 44019
2026-09-29 03:24:02.253 UTC [7273] FATAL: database "slave" does not exist
127.27.68.84:44019 - accepting connections
I20260929 03:24:02.254300 27921 mini_ranger.cc:161] Starting Ranger...
I20260929 03:24:02.269536 27921 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:24:02.735030 27921 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:24:02.735138 27921 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin
2026-09-29 03:24:03,022 [I] DB FLAVOR :POSTGRES
2026-09-29 03:24:03,023 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:24:03,023 [I] Checking connection..
2026-09-29 03:24:03,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:24:03,286 [I] Checking connection passed.
2026-09-29 03:24:03,287 [I] --------- Verifying version history table ---------
2026-09-29 03:24:03,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:24:03,523 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:24:03,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:24:03,759 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:24:03,759 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:24:03,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:24:03.992 UTC [7337] WARNING: there is no transaction in progress
2026-09-29 03:24:03,998 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:24:03,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:24:04,267 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:24:04,267 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:24:04,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:24:04,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:24:04,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fzwk.c.gcp-upstream.internal','N') ;"
2026-09-29 03:24:05,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:24:05,271 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:24:05,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:24:05,514 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:24:05,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:24:05,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:24:06,011 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:24:06,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:24:06.963 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:06.979 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:06.998 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.014 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.032 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.053 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.107 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.114 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.128 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.138 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.149 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.163 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.173 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.180 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.187 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07.194 UTC [7445] WARNING: there is no transaction in progress
2026-09-29 03:24:07,839 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:24:07,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:24:08,097 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:24:08,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:24:08,366 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:24:08,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:24:08,615 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:24:08,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:24:08,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:24:09,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fzwk.c.gcp-upstream.internal';"
2026-09-29 03:24:09,328 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:24:09,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:24:09,563 [I] DB_PATCHES have already been applied
I20260929 03:24:09.569496 27921 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko51MCN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:24:09.569561 27921 mini_ranger.cc:193] Using host: 127.27.68.84
I20260929 03:24:09.572605 27921 mini_ranger.cc:227] Ranger admin URL: http://127.27.68.84:37217
Sep 29, 2026 3:24:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:24:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:24:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:24:10.682737 27921 mini_ranger.cc:160] Time spent starting Ranger: real 8.428s user 0.003s sys 0.006s
Sep 29, 2026 3:24:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:24:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:24:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:24:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:24:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:24:33.312496 27921 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.68.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.68.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangerkms_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.68.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/rangerkms_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.68.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.68.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.68.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/HTTP_127.27.68.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:24:33 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:24:33.407362 27921 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:24:33.417184 27921 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:24:34.243206 27921 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:24:34.243317 27921 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms
I20260929 03:24:34.243340 27921 mini_ranger_kms.cc:227] Using postgres at 127.27.68.84:44019
2026-09-29 03:24:34,407 [I] DB FLAVOR :POSTGRES
2026-09-29 03:24:34,407 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:24:34,407 [I] Checking connection
2026-09-29 03:24:34,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:24:34,669 [I] connection success
2026-09-29 03:24:34,669 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:24:34,669 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:24:34,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:24:34,938 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:24:34,938 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:24:34,938 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:24:34,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.68.84:44019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:24:35,198 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:24:35.676132 27921 mini_ranger_kms.cc:310] Created kms service
I20260929 03:24:35.903785 27921 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:24:35.951406 27921 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:24:35 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:36.388331 27921 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:24:36.388540 27921 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko51MCN/build/release/bin/postgresql.jar:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko51MCN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:24:36.388561 27921 mini_ranger_kms.cc:240] Using host: 127.27.68.84
I20260929 03:24:36.392513 27921 mini_ranger_kms.cc:276] Ranger KMS PID: 7625
I20260929 03:24:36.392606 27921 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.68.84:45645
Sep 29, 2026 3:24:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:24:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:24:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:24:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:24:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:24:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:24:38.717746 27921 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.310s user 0.010s sys 0.137s
I20260929 03:24:38.717866 27921 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:24:38.717934 27921 mini_ranger_kms.cc:401] 127.27.68.84:45645/kms/v1/keys
Sep 29 03:24:38 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:24:39 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
Sep 29 03:24:39 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.68.84@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
Sep 29 03:24:42 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:24:42.090305 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46045
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46045
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:42.172351 7676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:42.172585 7676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:42.172611 7676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:42.173772 7676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:42.173810 7676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:42.173823 7676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:42.173835 7676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:42.173851 7676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:42.175477 7676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.68.84:45645/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46045
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:42.175662 7676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:42.175837 7676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:42.180652 7682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:42.180688 7681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:42.180775 7676 server_base.cc:1034] running on GCE node
W20260929 03:24:42.180649 7684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:42.181110 7676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:42.181429 7676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:42.182572 7676 hybrid_clock.cc:648] HybridClock initialized: now 1790652282182566 us; error 35 us; skew 500 ppm
Sep 29 03:24:42 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:42.190899 7676 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:42.191450 7676 webserver.cc:460] Webserver started at http://127.27.68.126:33801/ using document root <none> and password file <none>
I20260929 03:24:42.191704 7676 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:42.191766 7676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:42.191922 7676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:24:42 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:42.245878 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "401e95012cf7407b9f7bf7f461f3115c"
format_stamp: "Formatted at 2026-09-29 03:24:42 on dist-test-slave-fzwk"
server_key: "8094c6536e7f2c8c0fd6e486fe3ba05d"
server_key_iv: "cd5306ba7981de347159929e2bd47264"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:42.246404 7676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "401e95012cf7407b9f7bf7f461f3115c"
format_stamp: "Formatted at 2026-09-29 03:24:42 on dist-test-slave-fzwk"
server_key: "8094c6536e7f2c8c0fd6e486fe3ba05d"
server_key_iv: "cd5306ba7981de347159929e2bd47264"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:42.248399 7676 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260929 03:24:42.263396 7691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:42.263622 7676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:42.263718 7676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "401e95012cf7407b9f7bf7f461f3115c"
format_stamp: "Formatted at 2026-09-29 03:24:42 on dist-test-slave-fzwk"
server_key: "8094c6536e7f2c8c0fd6e486fe3ba05d"
server_key_iv: "cd5306ba7981de347159929e2bd47264"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:42.263816 7676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:42.279008 7676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:42.281330 7676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:42.281505 7676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:42.287012 7676 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46045
I20260929 03:24:42.287035 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46045 every 8 connection(s)
I20260929 03:24:42.287540 7676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:42.288264 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:42.290531 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Bootstrap starting.
I20260929 03:24:42.291134 7744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:42.291458 7744 log.cc:824] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:42.292212 7744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: No bootstrap required, opened a new log
I20260929 03:24:42.294235 7744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:42.294417 7744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:42.294469 7744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401e95012cf7407b9f7bf7f461f3115c, State: Initialized, Role: FOLLOWER
I20260929 03:24:42.294642 7744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:42.294744 7744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:42.294787 7744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:42.294835 7744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:42.295506 7744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:42.295624 7744 leader_election.cc:302] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401e95012cf7407b9f7bf7f461f3115c; no voters:
I20260929 03:24:42.295802 7744 leader_election.cc:288] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:24:42.295862 7747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:42.296073 7744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:42.296104 7747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 LEADER]: Becoming Leader. State: Replica: 401e95012cf7407b9f7bf7f461f3115c, State: Running, Role: LEADER
I20260929 03:24:42.296334 7747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:42.296540 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 7676
I20260929 03:24:42.296666 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:24:42.296751 7748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "401e95012cf7407b9f7bf7f461f3115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } } }
I20260929 03:24:42.296820 7748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:42.296902 27921 external_mini_cluster.cc:1428] Setting key aabeec79445506a625fcceacd4118a77
I20260929 03:24:42.296993 7749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401e95012cf7407b9f7bf7f461f3115c. Latest consensus state: current_term: 1 leader_uuid: "401e95012cf7407b9f7bf7f461f3115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } } }
I20260929 03:24:42.297046 7749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:42.297717 7744 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:24:42.299957 7756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:42.300925 7756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:42.303654 7756 catalog_manager.cc:1347] Generated new cluster ID: 725f44ba13a84b6db31cfc3997fa25b0
I20260929 03:24:42.303772 7756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:24:42 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:42.312003 7756 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:24:42.312855 7756 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:42.312911 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:42.297563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.27.68.126:46045)
0929 03:24:42.297788 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:42.297799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:42.298007 (+ 208us) server_negotiation.cc:407] Connection header received
0929 03:24:42.298141 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:42.298149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:42.299456 (+ 1307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:42.300569 (+ 1113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:42.300817 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:42.302882 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:42.304221 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:42.304852 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:42.307963 (+ 3111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:42.308068 (+ 105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:42.308072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:42.308072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:42.308083 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:42.310245 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:42.310691 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:42.310694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:42.310695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:42.310812 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:42.311066 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:42.311067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:42.311067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:42.311377 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:42.312149 (+ 772us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:42.312222 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:24:42.312354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260929 03:24:42.323582 7756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Generated new TSK 0
I20260929 03:24:42.323872 7756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:24:43.110086 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:43.200320 7789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:43.200572 7789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:43.200611 7789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:43.201937 7789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:43.202009 7789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:43.202056 7789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:24:43.203749 7789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:43.204030 7789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:43.204273 7789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:43.209440 7795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.209458 7794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.209462 7797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.209900 7789 server_base.cc:1034] running on GCE node
I20260929 03:24:43.210115 7789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:43.210389 7789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:43.211557 7789 hybrid_clock.cc:648] HybridClock initialized: now 1790652283211541 us; error 36 us; skew 500 ppm
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:43.214522 7789 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:24:43.215082 7789 webserver.cc:460] Webserver started at http://127.27.68.65:37907/ using document root <none> and password file <none>
I20260929 03:24:43.215448 7789 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:43.215523 7789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:43.215716 7789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:43.230717 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "93187fe4cc2b465ba5e59963efbb8d5e"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "a382ed775eaae1db60967a5cc83290cd"
server_key_iv: "5fde7d1efcb14b8d10e590fed68f8aba"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.231187 7789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "93187fe4cc2b465ba5e59963efbb8d5e"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "a382ed775eaae1db60967a5cc83290cd"
server_key_iv: "5fde7d1efcb14b8d10e590fed68f8aba"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.235666 7789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 03:24:43.251127 7804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.251405 7789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:43.251526 7789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "93187fe4cc2b465ba5e59963efbb8d5e"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "a382ed775eaae1db60967a5cc83290cd"
server_key_iv: "5fde7d1efcb14b8d10e590fed68f8aba"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.251633 7789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:43.279999 7789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:43.282590 7789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:43.282770 7789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:43.283039 7789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:43.283535 7789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:43.283628 7789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.283674 7789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:43.283692 7789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.290825 7789 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:44783
I20260929 03:24:43.291404 7789 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:24:43.295933 7919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:44783 every 8 connection(s)
I20260929 03:24:43.296555 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 7789
I20260929 03:24:43.296674 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:24:43.296872 27921 external_mini_cluster.cc:1428] Setting key 89a8c75d7480cbf14abc5076e218bae7
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:24:43.312817 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.296980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40341 (local address 127.27.68.126:46045)
0929 03:24:43.302158 (+ 5178us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:43.302160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:43.302174 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:24:43.302205 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:43.302206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:43.302220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:43.302312 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:43.303265 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.304020 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.305608 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.305929 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.305997 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.308863 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:43.308865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:43.308865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:43.308874 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:43.310774 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.311408 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.311410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.311410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.311486 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.311848 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.311849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.311849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.312020 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:43.312117 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:43.312606 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 03:24:43.312710 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5137,"thread_start_us":90,"threads_started":1}
I20260929 03:24:43.313292 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.296961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.65:40341)
0929 03:24:43.297488 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:43.297537 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:43.298156 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:43.302497 (+ 4341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:43.302503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:43.302799 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:43.303109 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.303119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.304148 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.304150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.305487 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.305491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.306097 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.306098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.306248 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.306910 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:43.306921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:43.308737 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:43.310882 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.310892 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.310895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.311295 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.311588 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.311589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.311590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.311756 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.312153 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:43.312161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:43.312357 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 03:24:43.312560 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 03:24:43.312759 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
I20260929 03:24:43.313831 7920 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:43.313939 7920 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:43.314142 7920 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:43.316107 7708 ts_manager.cc:194] Registered new tserver with Master: 93187fe4cc2b465ba5e59963efbb8d5e (127.27.68.65:44783)
I20260929 03:24:43.317025 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:40341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:24:43.333706 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:43.455412 7926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:43.455722 7926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:43.455766 7926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:43.457739 7926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:43.457813 7926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:43.457858 7926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:24:43.460325 7926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:43.460793 7926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:43.461123 7926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 03:24:43.468168 7933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.468350 7926 server_base.cc:1034] running on GCE node
W20260929 03:24:43.468389 7936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.468170 7934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.468816 7926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:43.469178 7926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:43.470392 7926 hybrid_clock.cc:648] HybridClock initialized: now 1790652283470371 us; error 35 us; skew 500 ppm
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:43.474066 7926 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:24:43.474717 7926 webserver.cc:460] Webserver started at http://127.27.68.66:41663/ using document root <none> and password file <none>
I20260929 03:24:43.475039 7926 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:43.475108 7926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:43.475311 7926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:43.491499 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "f390ec907f6687ab38ea7911ab051334"
server_key_iv: "747549b96fa85443cb8e2eadb41190d6"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.492121 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "f390ec907f6687ab38ea7911ab051334"
server_key_iv: "747549b96fa85443cb8e2eadb41190d6"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.503655 7926 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260929 03:24:43.519866 7943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.520139 7926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:43.520226 7926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "f390ec907f6687ab38ea7911ab051334"
server_key_iv: "747549b96fa85443cb8e2eadb41190d6"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.520316 7926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:43.553038 7926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:43.557083 7926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:43.557322 7926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:43.557632 7926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:43.558069 7926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:43.558162 7926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.558238 7926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:43.558295 7926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.565459 7926 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:45877
I20260929 03:24:43.566076 7926 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:24:43.568426 8057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:45877 every 8 connection(s)
I20260929 03:24:43.573997 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 7926
I20260929 03:24:43.574116 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:24:43.574309 27921 external_mini_cluster.cc:1428] Setting key d9bac6ba554cad8112c0533b812f391e
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:43.581282 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.569405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:38693 (local address 127.27.68.126:46045)
0929 03:24:43.569453 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:43.569455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:43.570108 (+ 653us) server_negotiation.cc:407] Connection header received
0929 03:24:43.570860 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:43.570862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:43.570880 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:43.570914 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:43.571870 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.572661 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.574333 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.574684 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.574730 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.577594 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:43.577596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:43.577596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:43.577605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:43.579502 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.580090 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.580091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.580092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.580156 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.580447 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.580448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.580449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.580591 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:43.580638 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:43.581077 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 03:24:43.581178 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:24:43.582180 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.569104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.66:38693)
0929 03:24:43.569955 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:43.569999 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:43.570727 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:43.571126 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:43.571131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:43.571425 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:43.571753 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.571760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.572783 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.572784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.574233 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.574236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.574905 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.574906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.575068 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.575725 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:43.575735 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:43.577455 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:43.579636 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.579645 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.579648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.579974 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.580229 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.580230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.580230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.580373 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.580687 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:43.580692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:43.580888 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 03:24:43.581639 (+ 751us) client_negotiation.cc:240] Negotiation successful
0929 03:24:43.581843 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":95,"threads_started":1}
I20260929 03:24:43.582942 8058 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:43.583061 8058 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:43.583391 8058 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:43.583916 7708 ts_manager.cc:194] Registered new tserver with Master: a7c4ec77d8e541ac9498425d8c1d2d7c (127.27.68.66:45877)
I20260929 03:24:43.584424 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:38693
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:24:43.605578 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:43.729494 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:43.729774 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:43.729815 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:43.731069 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:43.731142 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:43.731187 8063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:24:43.732786 8063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:43.733078 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:43.733342 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:43.739334 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.739374 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.739595 8063 server_base.cc:1034] running on GCE node
W20260929 03:24:43.739549 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.740075 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:43.740376 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:43.741534 8063 hybrid_clock.cc:648] HybridClock initialized: now 1790652283741509 us; error 45 us; skew 500 ppm
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:43.744331 8063 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:24:43.744894 8063 webserver.cc:460] Webserver started at http://127.27.68.67:43499/ using document root <none> and password file <none>
I20260929 03:24:43.745157 8063 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:43.745224 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:43.745369 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:43.759596 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "3fd144683eb6465598eac20a8d133681"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "0fccca630d018a8f9b2196f3d721cd55"
server_key_iv: "dec2fe2244847cba68aa6f50d7575860"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.760159 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "3fd144683eb6465598eac20a8d133681"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "0fccca630d018a8f9b2196f3d721cd55"
server_key_iv: "dec2fe2244847cba68aa6f50d7575860"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.762935 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 03:24:43.781044 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.781916 8063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 03:24:43.782084 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "3fd144683eb6465598eac20a8d133681"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "0fccca630d018a8f9b2196f3d721cd55"
server_key_iv: "dec2fe2244847cba68aa6f50d7575860"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:43.782282 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:43.803908 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:43.806833 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:43.806998 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:43.807314 8063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:43.807773 8063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:43.807823 8063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.807855 8063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:43.807919 8063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:43.814757 8063 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:41129
I20260929 03:24:43.815047 8191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:41129 every 8 connection(s)
I20260929 03:24:43.815367 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:24:43.820204 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8063
I20260929 03:24:43.820339 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:24:43.820525 27921 external_mini_cluster.cc:1428] Setting key 25e6e049272ba0a5b10bbcd9fd0be77f
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:43.828104 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.816471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:34883 (local address 127.27.68.126:46045)
0929 03:24:43.816518 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:43.816521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:43.816780 (+ 259us) server_negotiation.cc:407] Connection header received
0929 03:24:43.817661 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:43.817663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:43.817677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:43.817708 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:43.818566 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.819414 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.820940 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.821250 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.821295 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.824424 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:43.824426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:43.824426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:43.824435 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:43.826293 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.826834 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.826836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.826836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.826898 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:43.827240 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:43.827242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:43.827242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:43.827410 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:43.827499 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:43.827891 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 03:24:43.827990 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:24:43.828950 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:43.816475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.67:34883)
0929 03:24:43.816643 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:43.816693 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:43.817528 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:43.817892 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:43.817896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:43.818168 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:43.818446 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.818452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.819511 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.819512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.820815 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:43.820818 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:43.821407 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:43.821408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:43.821558 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:43.822198 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:43.822209 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:43.824315 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:43.826404 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.826413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.826416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.826739 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.826967 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:43.826967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:43.826968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:43.827118 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:43.827506 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:43.827514 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:43.827705 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 03:24:43.828497 (+ 792us) client_negotiation.cc:240] Negotiation successful
0929 03:24:43.828694 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260929 03:24:43.829720 8192 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:43.829840 8192 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:43.830075 8192 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:43.830540 7708 ts_manager.cc:194] Registered new tserver with Master: 3fd144683eb6465598eac20a8d133681 (127.27.68.67:41129)
I20260929 03:24:43.831055 7708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:34883
I20260929 03:24:43.832113 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:24:43.833024 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 7789
I20260929 03:24:43.840751 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 7926
I20260929 03:24:43.847921 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8063
I20260929 03:24:43.854447 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 7676
I20260929 03:24:43.859489 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46045
--webserver_interface=127.27.68.126
--webserver_port=33801
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46045
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:43.942682 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:43.942904 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:43.942927 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:43.944193 8195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:43.944238 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:43.944252 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:43.944264 8195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:43.944314 8195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:43.945897 8195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46045
--ranger_config_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:43.946153 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:43.946405 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:43.951253 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.951268 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:43.951315 8200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:43.951444 8195 server_base.cc:1034] running on GCE node
I20260929 03:24:43.951828 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:43.952147 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:43.953320 8195 hybrid_clock.cc:648] HybridClock initialized: now 1790652283953312 us; error 38 us; skew 500 ppm
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:43.955847 8195 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:43.956350 8195 webserver.cc:460] Webserver started at http://127.27.68.126:33801/ using document root <none> and password file <none>
I20260929 03:24:43.956580 8195 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:43.956660 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:24:43 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:44.008746 8195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:44.009752 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.009992 8195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:44.010097 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "401e95012cf7407b9f7bf7f461f3115c"
format_stamp: "Formatted at 2026-09-29 03:24:42 on dist-test-slave-fzwk"
server_key: "8094c6536e7f2c8c0fd6e486fe3ba05d"
server_key_iv: "cd5306ba7981de347159929e2bd47264"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:44.010618 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:44.026036 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:44.028505 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:44.028702 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:44.033247 8195 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46045
I20260929 03:24:44.033285 8263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46045 every 8 connection(s)
I20260929 03:24:44.033735 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:44.035075 8264 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:24:44.035360 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8195
I20260929 03:24:44.035687 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:44783
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:46045
--webserver_port=37907
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:44.036260 8264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Bootstrap starting.
I20260929 03:24:44.044744 8264 log.cc:824] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:44.046972 8264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:24:44.047348 8264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Bootstrap complete.
I20260929 03:24:44.049948 8264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:44.050196 8264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401e95012cf7407b9f7bf7f461f3115c, State: Initialized, Role: FOLLOWER
I20260929 03:24:44.050379 8264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:44.050447 8264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:44.050509 8264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:44.050566 8264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:24:44.051714 8264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:44.051836 8264 leader_election.cc:302] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 401e95012cf7407b9f7bf7f461f3115c; no voters:
I20260929 03:24:44.052155 8264 leader_election.cc:288] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:24:44.052191 8268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:24:44.052299 8268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [term 2 LEADER]: Becoming Leader. State: Replica: 401e95012cf7407b9f7bf7f461f3115c, State: Running, Role: LEADER
I20260929 03:24:44.052438 8264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:44.052454 8268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } }
I20260929 03:24:44.052865 8270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401e95012cf7407b9f7bf7f461f3115c. Latest consensus state: current_term: 2 leader_uuid: "401e95012cf7407b9f7bf7f461f3115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } } }
I20260929 03:24:44.052943 8270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:44.053342 8274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:44.053376 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "401e95012cf7407b9f7bf7f461f3115c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "401e95012cf7407b9f7bf7f461f3115c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46045 } } }
I20260929 03:24:44.053442 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:44.053871 8274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:44.054003 8274 catalog_manager.cc:1259] Loaded cluster ID: 725f44ba13a84b6db31cfc3997fa25b0
I20260929 03:24:44.054033 8274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:24:44.054277 8264 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:24:44.057684 8274 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:44.058375 8274 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Loaded TSK: 0
I20260929 03:24:44.058697 8274 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:24:44.150432 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:44.150848 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:44.150961 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:44.152889 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:44.153010 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:44.153115 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:24:44.155661 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:44783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=37907
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:44.156234 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:44.156600 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:44.163127 8302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:44.163127 8304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:44.163481 8301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:44.163971 8266 server_base.cc:1034] running on GCE node
I20260929 03:24:44.164198 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:44.164494 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:44.165685 8266 hybrid_clock.cc:648] HybridClock initialized: now 1790652284165651 us; error 47 us; skew 500 ppm
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:44.173053 8266 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:24:44.173794 8266 webserver.cc:460] Webserver started at http://127.27.68.65:37907/ using document root <none> and password file <none>
I20260929 03:24:44.174145 8266 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:44.174299 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:44.197582 8266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:24:44.198676 8312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.198889 8266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:44.198998 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "93187fe4cc2b465ba5e59963efbb8d5e"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "a382ed775eaae1db60967a5cc83290cd"
server_key_iv: "5fde7d1efcb14b8d10e590fed68f8aba"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:44.199643 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:44.218077 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:44.220779 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:44.220953 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:44.221189 8266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:44.221611 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:44.221660 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.221690 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:44.221741 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.232332 8266 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:44783
I20260929 03:24:44.232434 8425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:44783 every 8 connection(s)
I20260929 03:24:44.232960 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:24:44.233294 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8266
I20260929 03:24:44.233615 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:45877
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:46045
--webserver_port=41663
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:44.245309 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.233817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.65:46957)
0929 03:24:44.234098 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:44.234147 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:44.234883 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:44.235509 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:44.235513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:44.235753 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:44.236038 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.236045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.237422 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.237423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.238325 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.238328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.238957 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.238959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.239166 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.239867 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:44.239879 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:44.241375 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:44.243505 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.243512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.243515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.243753 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.244022 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.244023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.244023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.244161 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.244620 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:44.244627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:44.244779 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:24:44.244893 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:24:44.245074 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
I20260929 03:24:44.245296 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.234071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:46957 (local address 127.27.68.126:46045)
0929 03:24:44.234329 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:44.234339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:44.234380 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:24:44.235210 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:44.235270 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:44.235326 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:44.235467 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:44.236192 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.237299 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.238429 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.238850 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.238971 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.241497 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:44.241501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:44.241501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:44.241514 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:44.243409 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.243844 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.243848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.243848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.243952 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.244241 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.244242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.244242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.244529 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:44.244629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:44.244952 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:24:44.245070 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260929 03:24:44.246460 8426 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:44.246562 8426 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.246726 8426 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.247654 8426 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:24:44.247728 8426 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.247798 8426 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:44.248098 8426 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.248171 8426 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.248440 8426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:24:44.356555 8430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:44.356861 8430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:44.356913 8430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:44.358843 8430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:44.358959 8430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:44.359055 8430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:24:44.361462 8430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:45877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=41663
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:44.361735 8430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:44.362087 8430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:44.369362 8438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:44.369531 8430 server_base.cc:1034] running on GCE node
W20260929 03:24:44.371388 8435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:44.371379 8436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:44.374647 8430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:44.375039 8430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:44.376248 8430 hybrid_clock.cc:648] HybridClock initialized: now 1790652284376217 us; error 48 us; skew 500 ppm
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:44.379922 8430 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:24:44.380555 8430 webserver.cc:460] Webserver started at http://127.27.68.66:41663/ using document root <none> and password file <none>
I20260929 03:24:44.380877 8430 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:44.380970 8430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:44.400417 8430 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260929 03:24:44.401777 8446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.402009 8430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:44.402100 8430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "f390ec907f6687ab38ea7911ab051334"
server_key_iv: "747549b96fa85443cb8e2eadb41190d6"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:44.402607 8430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:44.417615 8430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:44.419911 8430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:44.420092 8430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:44.420373 8430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:44.420780 8430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:44.420842 8430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.420885 8430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:44.420912 8430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.428164 8430 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:45877
I20260929 03:24:44.428743 8430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:24:44.429574 8559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:45877 every 8 connection(s)
I20260929 03:24:44.430516 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8430
I20260929 03:24:44.430846 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.68.84:45645/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:41129
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:46045
--webserver_port=43499
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:44.441417 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.430066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:34847 (local address 127.27.68.126:46045)
0929 03:24:44.430118 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:44.430121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:44.430132 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:24:44.430725 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:44.430727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:44.430743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:44.430814 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:44.431911 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.432724 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.434331 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.434694 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.434747 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.437596 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:44.437598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:44.437598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:44.437607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:44.439546 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.440136 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.440137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.440138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.440205 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.440537 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.440538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.440539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.440710 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:44.440762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:44.441201 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 03:24:44.441295 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:24:44.441680 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.429713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.66:34847)
0929 03:24:44.429894 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:44.429932 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:44.430607 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:44.430892 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:44.430896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:44.431511 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:44.431812 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.431819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.432844 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.432845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.434220 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.434224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.434874 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.434876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.435042 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.435647 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:44.435659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:44.437473 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:44.439656 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.439664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.439666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.440028 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.440289 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.440290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.440290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.440450 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.440819 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:44.440826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:44.441021 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 03:24:44.441154 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:24:44.441345 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260929 03:24:44.442402 8560 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:44.442525 8560 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.442766 8560 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.443514 8560 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:24:44.443593 8560 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.443655 8560 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:44.444099 8560 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.444180 8560 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.444572 8560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:24:44.536144 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:44.536527 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:44.536608 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:44.538450 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:44.538513 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:44.538553 8563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:24:44.540854 8563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.68.84:45645/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:41129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=43499
--tserver_master_addrs=127.27.68.126:46045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:44.541145 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:44.541450 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:44.547492 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:44.547492 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:44.547739 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:44.548327 8563 server_base.cc:1034] running on GCE node
I20260929 03:24:44.548676 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:44.548969 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:44.552269 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790652284552232 us; error 53 us; skew 500 ppm
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:44.555954 8563 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:24:44.556574 8563 webserver.cc:460] Webserver started at http://127.27.68.67:43499/ using document root <none> and password file <none>
I20260929 03:24:44.556946 8563 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:44.557093 8563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:44.573616 8563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:44.574819 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.575075 8563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:44.575188 8563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "3fd144683eb6465598eac20a8d133681"
format_stamp: "Formatted at 2026-09-29 03:24:43 on dist-test-slave-fzwk"
server_key: "0fccca630d018a8f9b2196f3d721cd55"
server_key_iv: "dec2fe2244847cba68aa6f50d7575860"
server_key_version: "kuduclusterkey@0"
I20260929 03:24:44.575836 8563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:44.623706 8563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:44.625723 8563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:44.625876 8563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:44.626120 8563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:44.626509 8563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:44.626551 8563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.626581 8563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:44.626598 8563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:44.633426 8563 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:41129
I20260929 03:24:44.633522 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:41129 every 8 connection(s)
I20260929 03:24:44.633868 8563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:24:44.639305 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8563
Sep 29 03:24:44 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:44.645053 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.634663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:39703 (local address 127.27.68.126:46045)
0929 03:24:44.634736 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:44.634738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:44.634974 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:24:44.635590 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:44.635592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:44.635605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:44.635664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:44.636247 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.636750 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.637922 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.638217 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.638292 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.641425 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:44.641427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:44.641427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:44.641436 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:44.643175 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.643761 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.643763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.643763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.643828 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.644209 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.644210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.644211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.644350 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:44.644397 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:44.644842 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 03:24:44.644922 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:24:44.645720 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.634653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46045 (local address 127.27.68.67:39703)
0929 03:24:44.634838 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:44.634877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:44.635467 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:44.635750 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:44.635754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:44.635940 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:44.636148 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.636153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.636852 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.636853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.637812 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:44.637816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.638356 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.638358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:44.638520 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.639082 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:44.639093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:44.641286 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:44.643322 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.643337 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.643339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.643676 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.643921 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:44.643922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:44.643923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:44.644112 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:44.644450 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:44.644457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:44.644655 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 03:24:44.645229 (+ 574us) client_negotiation.cc:240] Negotiation successful
0929 03:24:44.645415 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260929 03:24:44.646237 8694 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46045
I20260929 03:24:44.646353 8694 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.646575 8694 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.647267 8694 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:46045 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:24:44.647339 8694 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.647401 8694 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:44.647979 8694 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:44.648057 8694 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
W20260929 03:24:44.648394 8694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:24:44.650130 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:44.639752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.27.68.126:46045)
0929 03:24:44.640101 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:44.640103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:44.640110 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:24:44.640133 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:44.640134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:44.640145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:44.640214 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:44.641025 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.641769 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.643343 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:44.643547 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:44.643618 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:44.645085 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:44.645086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:44.645087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:44.645094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:44.648695 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.649141 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.649143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.649143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.649214 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:44.649522 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:44.649523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:44.649523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:44.649666 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:44.649842 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:44.649932 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:24:44.650029 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":86,"threads_started":1}
Sep 29 03:24:45 dist-test-slave-fzwk krb5kdc[7232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:45.248651 8426 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:45.248888 8426 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:45.249315 8228 ts_manager.cc:194] Registered new tserver with Master: 93187fe4cc2b465ba5e59963efbb8d5e (127.27.68.65:44783)
I20260929 03:24:45.250360 8228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:46957
I20260929 03:24:45.444793 8560 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:45.444983 8560 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:45.445434 8228 ts_manager.cc:194] Registered new tserver with Master: a7c4ec77d8e541ac9498425d8c1d2d7c (127.27.68.66:45877)
I20260929 03:24:45.446028 8228 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:34847
Sep 29 03:24:45 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:45.648563 8694 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:45.648737 8694 heartbeater.cc:511] Master 127.27.68.126:46045 requested a full tablet report, sending...
I20260929 03:24:45.649088 8227 ts_manager.cc:194] Registered new tserver with Master: 3fd144683eb6465598eac20a8d133681 (127.27.68.67:41129)
I20260929 03:24:45.649796 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:39703
I20260929 03:24:45.652164 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:24:45.661659 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.653324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.27.68.126:46045)
0929 03:24:45.653612 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.653614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.653791 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:24:45.653818 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.653819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.653841 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.653897 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:45.654672 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.655457 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.656412 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.656685 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.656748 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.658312 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:45.658314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:45.658315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:45.658322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:45.660147 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:45.660685 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:45.660687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:45.660687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:45.660759 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:45.660963 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:45.660964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:45.660964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:45.661105 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:45.661195 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.661444 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.661549 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":70,"threads_started":1}
I20260929 03:24:45.664181 8227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:24:45.666503 8227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:24:45.677219 8721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.669703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:45877 (local address 127.0.0.1:43192)
0929 03:24:45.671360 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.671408 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:45.671479 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:45.671778 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:45.671781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:45.671784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:45.671860 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.671862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.673397 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.673398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.674743 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.674746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.676887 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.676888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.676982 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.676999 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:24:45.677049 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:24:45.677106 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"thread_start_us":79,"threads_started":1}
I20260929 03:24:45.677706 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.669920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:41129 (local address 127.0.0.1:50582)
0929 03:24:45.671576 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.671583 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:45.671614 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:45.671874 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:45.671875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:45.671877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:45.671937 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.671938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.673499 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.673500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.675863 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.675865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.676929 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.676931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.677017 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.677024 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:24:45.677059 (+ 35us) client_negotiation.cc:240] Negotiation successful
0929 03:24:45.677112 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1643,"mutex_wait_us":1208,"thread_start_us":64,"threads_started":1}
I20260929 03:24:45.677788 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.670049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.27.68.67:41129)
0929 03:24:45.671714 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.671718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.671728 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:24:45.671761 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.671763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.671777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.671999 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:45.672365 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.673394 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.675962 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.676789 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.677495 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.677597 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.677629 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.677684 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":63,"threads_started":1}
I20260929 03:24:45.678226 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.670197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.27.68.66:45877)
0929 03:24:45.671609 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.671615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.671625 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:24:45.671661 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.671663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.671680 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.671929 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:45.672012 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.673293 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.675996 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.676780 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.678009 (+ 1229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.678082 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.678109 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.678152 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":51,"threads_started":1}
I20260929 03:24:45.678785 8628 tablet_service.cc:1467] Processing CreateTablet for tablet f0022194ab3244aeab8787e88d026268 (DEFAULT_TABLE table=test-table [id=37eb786a3e0643f6a1570c5f9a11fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:45.679204 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0022194ab3244aeab8787e88d026268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:45.679124 8494 tablet_service.cc:1467] Processing CreateTablet for tablet f0022194ab3244aeab8787e88d026268 (DEFAULT_TABLE table=test-table [id=37eb786a3e0643f6a1570c5f9a11fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:45.679555 8494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0022194ab3244aeab8787e88d026268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:45.682214 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.669704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44783 (local address 127.0.0.1:44966)
0929 03:24:45.671362 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.671412 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:45.671479 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:45.678713 (+ 7234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:45.678715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:45.678717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:45.678758 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.678760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.680019 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.680020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.681040 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.681042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.681994 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.681995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.682055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.682061 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:24:45.682093 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:24:45.682135 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"thread_start_us":108,"threads_started":1}
I20260929 03:24:45.682644 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.670326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.27.68.65:44783)
0929 03:24:45.678516 (+ 8190us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.678521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.678532 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:24:45.678570 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.678572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.678590 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.678826 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:45.678903 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.679910 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.681147 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.681873 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.682426 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.682496 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.682525 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.682566 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8151,"thread_start_us":58,"threads_started":1}
I20260929 03:24:45.683305 8360 tablet_service.cc:1467] Processing CreateTablet for tablet f0022194ab3244aeab8787e88d026268 (DEFAULT_TABLE table=test-table [id=37eb786a3e0643f6a1570c5f9a11fd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:45.683676 8360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0022194ab3244aeab8787e88d026268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:45.686934 8727 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Bootstrap starting.
I20260929 03:24:45.687956 8727 tablet_bootstrap.cc:654] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:45.688304 8727 log.cc:824] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:45.688690 8729 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Bootstrap starting.
I20260929 03:24:45.689240 8727 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: No bootstrap required, opened a new log
I20260929 03:24:45.689319 8727 ts_tablet_manager.cc:1397] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:24:45.691905 8729 tablet_bootstrap.cc:654] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:45.692085 8727 raft_consensus.cc:348] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.692312 8729 log.cc:824] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:45.692382 8727 raft_consensus.cc:374] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:45.692487 8727 raft_consensus.cc:729] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd144683eb6465598eac20a8d133681, State: Initialized, Role: FOLLOWER
I20260929 03:24:45.692692 8727 consensus_queue.cc:260] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.693073 8727 ts_tablet_manager.cc:1428] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 03:24:45.693106 8728 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Bootstrap starting.
I20260929 03:24:45.693238 8694 heartbeater.cc:503] Master 127.27.68.126:46045 was elected leader, sending a full tablet report...
I20260929 03:24:45.693813 8728 tablet_bootstrap.cc:654] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:45.694092 8728 log.cc:824] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:45.694859 8729 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: No bootstrap required, opened a new log
I20260929 03:24:45.694928 8729 ts_tablet_manager.cc:1397] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260929 03:24:45.695600 8728 tablet_bootstrap.cc:492] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: No bootstrap required, opened a new log
I20260929 03:24:45.695667 8728 ts_tablet_manager.cc:1397] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:24:45.697455 8729 raft_consensus.cc:348] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.697618 8729 raft_consensus.cc:374] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:45.697645 8729 raft_consensus.cc:729] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93187fe4cc2b465ba5e59963efbb8d5e, State: Initialized, Role: FOLLOWER
I20260929 03:24:45.697801 8729 consensus_queue.cc:260] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.697779 8728 raft_consensus.cc:348] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.697950 8728 raft_consensus.cc:374] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:45.697986 8728 raft_consensus.cc:729] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c4ec77d8e541ac9498425d8c1d2d7c, State: Initialized, Role: FOLLOWER
I20260929 03:24:45.698057 8729 ts_tablet_manager.cc:1428] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:24:45.698092 8728 consensus_queue.cc:260] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.698170 8426 heartbeater.cc:503] Master 127.27.68.126:46045 was elected leader, sending a full tablet report...
I20260929 03:24:45.698354 8728 ts_tablet_manager.cc:1428] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:24:45.698419 8560 heartbeater.cc:503] Master 127.27.68.126:46045 was elected leader, sending a full tablet report...
W20260929 03:24:45.734761 8427 tablet.cc:2369] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:24:45.794376 8734 raft_consensus.cc:482] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:45.794541 8734 raft_consensus.cc:504] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.794942 8734 leader_election.cc:288] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7c4ec77d8e541ac9498425d8c1d2d7c (127.27.68.66:45877), 3fd144683eb6465598eac20a8d133681 (127.27.68.67:41129)
I20260929 03:24:45.799849 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.795516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:60559 (local address 127.27.68.66:45877)
0929 03:24:45.795564 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.795566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.796185 (+ 619us) server_negotiation.cc:407] Connection header received
0929 03:24:45.796294 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.796296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.796306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.796351 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:45.796580 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.797292 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.798843 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.799397 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.799456 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.799467 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.799718 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.799769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:24:45.800451 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.795610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:45877 (local address 127.27.68.65:60559)
0929 03:24:45.796112 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.796124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:45.796215 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:45.796424 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:45.796425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:45.796427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:45.796506 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.796508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.797422 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.797423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.798709 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.798714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.799549 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.799550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.799623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.799627 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:24:45.800281 (+ 654us) client_negotiation.cc:240] Negotiation successful
0929 03:24:45.800348 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":94,"threads_started":1}
I20260929 03:24:45.801029 8498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0022194ab3244aeab8787e88d026268" candidate_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" is_pre_election: true
I20260929 03:24:45.801225 8498 raft_consensus.cc:2454] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93187fe4cc2b465ba5e59963efbb8d5e in term 0.
I20260929 03:24:45.801527 8314 leader_election.cc:302] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93187fe4cc2b465ba5e59963efbb8d5e, a7c4ec77d8e541ac9498425d8c1d2d7c; no voters:
I20260929 03:24:45.801676 8734 raft_consensus.cc:2781] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:24:45.801723 8734 raft_consensus.cc:482] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:24:45.801746 8734 raft_consensus.cc:3037] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:45.802701 8734 raft_consensus.cc:504] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.802845 8734 leader_election.cc:288] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [CANDIDATE]: Term 1 election: Requested vote from peers a7c4ec77d8e541ac9498425d8c1d2d7c (127.27.68.66:45877), 3fd144683eb6465598eac20a8d133681 (127.27.68.67:41129)
I20260929 03:24:45.803086 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.796009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:39723 (local address 127.27.68.67:41129)
0929 03:24:45.796053 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.796055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.797464 (+ 1409us) server_negotiation.cc:407] Connection header received
0929 03:24:45.799906 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.799908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.799922 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.799954 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:45.800182 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.800839 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.802143 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.802684 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.802737 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.802748 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.802973 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.803016 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:24:45.803187 8498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0022194ab3244aeab8787e88d026268" candidate_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c"
I20260929 03:24:45.803431 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.795837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:41129 (local address 127.27.68.65:39723)
0929 03:24:45.797353 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.797367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:45.797651 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:45.800027 (+ 2376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:45.800028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:45.800031 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:45.800110 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.800111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.800958 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.800959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.802033 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:45.802037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.802811 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.802812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:45.802888 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.802892 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:24:45.803297 (+ 405us) client_negotiation.cc:240] Negotiation successful
0929 03:24:45.803342 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"thread_start_us":60,"threads_started":1}
I20260929 03:24:45.803426 8498 raft_consensus.cc:3037] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:45.804370 8498 raft_consensus.cc:2454] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93187fe4cc2b465ba5e59963efbb8d5e in term 1.
I20260929 03:24:45.804709 8314 leader_election.cc:302] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93187fe4cc2b465ba5e59963efbb8d5e, a7c4ec77d8e541ac9498425d8c1d2d7c; no voters:
I20260929 03:24:45.805044 8734 raft_consensus.cc:2781] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:45.805063 8648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0022194ab3244aeab8787e88d026268" candidate_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd144683eb6465598eac20a8d133681" is_pre_election: true
I20260929 03:24:45.805209 8734 raft_consensus.cc:686] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [term 1 LEADER]: Becoming Leader. State: Replica: 93187fe4cc2b465ba5e59963efbb8d5e, State: Running, Role: LEADER
I20260929 03:24:45.805225 8648 raft_consensus.cc:2454] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93187fe4cc2b465ba5e59963efbb8d5e in term 0.
I20260929 03:24:45.805311 8734 consensus_queue.cc:237] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } }
I20260929 03:24:45.805359 8647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0022194ab3244aeab8787e88d026268" candidate_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd144683eb6465598eac20a8d133681"
I20260929 03:24:45.805428 8647 raft_consensus.cc:3037] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:45.806339 8647 raft_consensus.cc:2454] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93187fe4cc2b465ba5e59963efbb8d5e in term 1.
I20260929 03:24:45.806210 8228 catalog_manager.cc:5495] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 93187fe4cc2b465ba5e59963efbb8d5e (127.27.68.65). New cstate: current_term: 1 leader_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93187fe4cc2b465ba5e59963efbb8d5e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 44783 } health_report { overall_health: HEALTHY } } }
I20260929 03:24:45.824390 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.820789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.27.68.65:44783)
0929 03:24:45.820834 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:45.820837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:45.820985 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:24:45.821031 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:45.821033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:45.821056 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:45.821123 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:24:45.821329 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.822086 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.823180 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:45.823512 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:45.823582 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:45.823818 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:24:45.823985 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:24:45.824074 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:45.824280 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 03:24:45.824305 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 03:24:45 dist-test-slave-fzwk krb5kdc[7232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for HTTP/127.27.68.84@KRBTEST.COM
I20260929 03:24:45.827682 8647 raft_consensus.cc:1261] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 1 FOLLOWER]: Refusing update from remote peer 93187fe4cc2b465ba5e59963efbb8d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:45.828156 8738 consensus_queue.cc:1035] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd144683eb6465598eac20a8d133681" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 41129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:45.829998 8498 raft_consensus.cc:1261] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 1 FOLLOWER]: Refusing update from remote peer 93187fe4cc2b465ba5e59963efbb8d5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:45.830346 8738 consensus_queue.cc:1035] T f0022194ab3244aeab8787e88d026268 P 93187fe4cc2b465ba5e59963efbb8d5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4ec77d8e541ac9498425d8c1d2d7c" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:45.833336 8742 mvcc.cc:205] Tried to move back new op lower bound from 7334511762746634240 to 7334511762659217408. Current Snapshot: MvccSnapshot[applied={T|T < 7334511762746634240}]
I20260929 03:24:45.833992 8745 mvcc.cc:205] Tried to move back new op lower bound from 7334511762746634240 to 7334511762659217408. Current Snapshot: MvccSnapshot[applied={T|T < 7334511762746634240}]
I20260929 03:24:45.836256 8743 mvcc.cc:205] Tried to move back new op lower bound from 7334511762746634240 to 7334511762659217408. Current Snapshot: MvccSnapshot[applied={T|T < 7334511762746634240}]
I20260929 03:24:45.838652 8228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:45.838760 8228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:45.839756 8228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 401e95012cf7407b9f7bf7f461f3115c: Sending DeleteTablet for 3 replicas of tablet f0022194ab3244aeab8787e88d026268
I20260929 03:24:45.840122 8360 tablet_service.cc:1514] Processing DeleteTablet for tablet f0022194ab3244aeab8787e88d026268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:45 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:44966
I20260929 03:24:45.840732 8628 tablet_service.cc:1514] Processing DeleteTablet for tablet f0022194ab3244aeab8787e88d026268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:45 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50582
I20260929 03:24:45.841188 8494 tablet_service.cc:1514] Processing DeleteTablet for tablet f0022194ab3244aeab8787e88d026268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:45 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:43192
I20260929 03:24:45.841763 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8266
I20260929 03:24:45.850626 8753 tablet_replica.cc:330] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: stopping tablet replica
I20260929 03:24:45.850832 8752 tablet_replica.cc:330] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: stopping tablet replica
I20260929 03:24:45.850998 8752 raft_consensus.cc:2229] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:45.851147 8753 raft_consensus.cc:2229] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:45.851347 8752 raft_consensus.cc:2258] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:45.852595 8753 raft_consensus.cc:2258] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:45.853394 8752 ts_tablet_manager.cc:1905] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:24:45.858543 8215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:44783 (error 108)
I20260929 03:24:45.859122 8753 ts_tablet_manager.cc:1905] T f0022194ab3244aeab8787e88d026268 P a7c4ec77d8e541ac9498425d8c1d2d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:45.859452 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8430
I20260929 03:24:45.860075 8752 ts_tablet_manager.cc:1918] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:45.860159 8752 log.cc:1196] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/f0022194ab3244aeab8787e88d026268
I20260929 03:24:45.860473 8752 ts_tablet_manager.cc:1939] T f0022194ab3244aeab8787e88d026268 P 3fd144683eb6465598eac20a8d133681: Deleting consensus metadata
I20260929 03:24:45.867058 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.866856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:44783 (local address 127.0.0.1:44982)
0929 03:24:45.866949 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.867009 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:44783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 03:24:45.867138 8213 catalog_manager.cc:4841] TS 3fd144683eb6465598eac20a8d133681 (127.27.68.67:41129): tablet f0022194ab3244aeab8787e88d026268 (table test-table [id=37eb786a3e0643f6a1570c5f9a11fd92]) successfully deleted
W20260929 03:24:45.867372 8215 catalog_manager.cc:4584] TS 93187fe4cc2b465ba5e59963efbb8d5e (127.27.68.65:44783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0022194ab3244aeab8787e88d026268: Network error: Client connection negotiation failed: client connection to 127.27.68.65:44783: connect: Connection refused (error 111)
I20260929 03:24:45.867750 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:45.867664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:45877 (local address 127.0.0.1:43194)
0929 03:24:45.867704 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:45.867725 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.66:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:24:45.867949 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8563
I20260929 03:24:45.874589 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8195
I20260929 03:24:45.882742 27921 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:24:46.228693 27921 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:24:46.228786 27921 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:24:46.564998 27921 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:24:46.565 UTC [7262] LOG: received smart shutdown request
2026-09-29 03:24:46.569 UTC [7262] LOG: background worker "logical replication launcher" (PID 7271) exited with exit code 1
2026-09-29 03:24:46.569 UTC [7266] LOG: shutting down
2026-09-29 03:24:46.727 UTC [7262] LOG: database system is shut down
2026-09-29T03:24:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49298 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:24:46 dist-test-slave-fzwk krb5kdc[8764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:24:46 dist-test-slave-fzwk krb5kdc[8764](info): set up 2 sockets
Sep 29 03:24:46 dist-test-slave-fzwk krb5kdc[8764](info): commencing operation
krb5kdc: starting...
W20260929 03:24:47.871479 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:24:47 dist-test-slave-fzwk krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:24:47.992295 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:36839
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:48.073987 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:48.074251 8777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:48.074292 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:48.075500 8777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:48.075574 8777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:48.075598 8777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:48.075620 8777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:48.075650 8777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:48.077221 8777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:48.077517 8777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:48.077750 8777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:48.082612 8782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.082587 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.082670 8783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:48.082927 8777 server_base.cc:1034] running on GCE node
I20260929 03:24:48.083206 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:48.083570 8777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:48.084714 8777 hybrid_clock.cc:648] HybridClock initialized: now 1790652288084702 us; error 29 us; skew 500 ppm
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:48.091400 8777 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:48.091939 8777 webserver.cc:460] Webserver started at http://127.27.68.126:37169/ using document root <none> and password file <none>
I20260929 03:24:48.092193 8777 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:48.092242 8777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:48.092397 8777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:48.093528 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "94e954352ccd44408c66bb6892aa62b1"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "64378ae922c5ad5d3f7b644cf60588f1"
server_key_iv: "867eccae97b815caa7cb1475b1da6eb6"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.093904 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "94e954352ccd44408c66bb6892aa62b1"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "64378ae922c5ad5d3f7b644cf60588f1"
server_key_iv: "867eccae97b815caa7cb1475b1da6eb6"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.095346 8777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:24:48.096119 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:48.096308 8777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:48.096400 8777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "94e954352ccd44408c66bb6892aa62b1"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "64378ae922c5ad5d3f7b644cf60588f1"
server_key_iv: "867eccae97b815caa7cb1475b1da6eb6"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.096498 8777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:48.105475 8777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:48.107441 8777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:48.107600 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:48.111845 8777 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:36839
I20260929 03:24:48.111898 8844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:36839 every 8 connection(s)
I20260929 03:24:48.112304 8777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:48.113199 8845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:48.114866 8845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.117152 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8777
I20260929 03:24:48.117267 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:24:48.117460 27921 external_mini_cluster.cc:1428] Setting key 4e1da0c308ef877715514e66dc2fa2db
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.68.125@KRBTEST.COM; defaulting to no policy
I20260929 03:24:48.126389 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.115684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36839 (local address 127.0.0.1:40244)
0929 03:24:48.115870 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.115924 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.116442 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.116754 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.116757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.117073 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.117238 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.117244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.118826 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.118827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.120065 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.120069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.120582 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.120583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.120676 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.121234 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.121244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.122905 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.124809 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.124815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.124817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.125018 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.125276 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.125277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.125277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.125369 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.125749 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.125754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.125821 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.125936 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.126194 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":106,"thread_start_us":80,"threads_started":1}
I20260929 03:24:48.126391 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.115685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.27.68.126:36839)
0929 03:24:48.115897 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.115907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.116028 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:24:48.116626 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.116628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.116645 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.116747 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.117376 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.118708 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.120163 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.120488 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.120574 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.123016 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.123024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.123025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.123039 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.124699 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.125118 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.125121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.125122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.125205 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.125436 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.125437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.125438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.125666 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.125761 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.125963 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.126128 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":104,"threads_started":1}
I20260929 03:24:48.127281 8845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.127694 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.127572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37592)
0929 03:24:48.127624 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.127671 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:24:48.127842 8796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.125@KRBTEST.COM" created.
I20260929 03:24:48.128398 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.128307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37604)
0929 03:24:48.128353 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.128378 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:24:48.128528 8845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:34911: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
I20260929 03:24:48.142681 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:34911
--webserver_interface=127.27.68.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:48.177774 8845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } attempt: 1
I20260929 03:24:48.178321 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.178182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37616)
0929 03:24:48.178247 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.178291 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260929 03:24:48.178707 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.178626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37622)
0929 03:24:48.178667 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.178691 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:24:48.178839 8845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:34911: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
W20260929 03:24:48.224840 8852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:48.225051 8852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:48.225075 8852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:48.226163 8852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:48.226197 8852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:48.226212 8852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:48.226223 8852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:48.226235 8852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:48.227785 8852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.68.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:48.228024 8852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:48.228193 8852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:48.233017 8857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.233009 8860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.233009 8858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:48.233163 8852 server_base.cc:1034] running on GCE node
I20260929 03:24:48.233583 8852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:48.233848 8852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:48.234997 8852 hybrid_clock.cc:648] HybridClock initialized: now 1790652288234986 us; error 32 us; skew 500 ppm
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:48.237578 8852 init.cc:377] Logged in from keytab as kudu/127.27.68.125@KRBTEST.COM (short username kudu)
I20260929 03:24:48.238013 8852 webserver.cc:460] Webserver started at http://127.27.68.125:37327/ using document root <none> and password file <none>
I20260929 03:24:48.238261 8852 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:48.238332 8852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:48.238481 8852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:48.239609 8852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data/instance:
uuid: "6629454f0a874005a860583b03dd9f27"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "e2ea0915f66499ca722c6ba004d70b74"
server_key_iv: "cb681cae69e8cf117e0d2d2a1a521b61"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.239966 8852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal/instance:
uuid: "6629454f0a874005a860583b03dd9f27"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "e2ea0915f66499ca722c6ba004d70b74"
server_key_iv: "cb681cae69e8cf117e0d2d2a1a521b61"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.241326 8852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:48.242336 8867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:48.242508 8852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:48.242616 8852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "6629454f0a874005a860583b03dd9f27"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "e2ea0915f66499ca722c6ba004d70b74"
server_key_iv: "cb681cae69e8cf117e0d2d2a1a521b61"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.242705 8852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:48.247007 8845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } attempt: 2
I20260929 03:24:48.247531 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.247382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37628)
0929 03:24:48.247458 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.247499 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 03:24:48.247916 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.247827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37644)
0929 03:24:48.247872 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.247899 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:24:48.248027 8845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:34911: Network error: Client connection negotiation failed: client connection to 127.27.68.125:34911: connect: Connection refused (error 111)
I20260929 03:24:48.270428 8852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:48.272500 8852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:48.272660 8852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:48.276875 8852 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.125:34911
I20260929 03:24:48.276911 8919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.125:34911 every 8 connection(s)
I20260929 03:24:48.277309 8852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
I20260929 03:24:48.278067 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8852
I20260929 03:24:48.278172 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-1/wal/instance
I20260929 03:24:48.278291 8920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:48.278393 27921 external_mini_cluster.cc:1428] Setting key c8c0233fdc4eb3e05806418a2efd215e
I20260929 03:24:48.280520 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.68.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.124@KRBTEST.COM" created.
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:48.292564 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.281308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.27.68.126:36839)
0929 03:24:48.281378 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.281380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.281595 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:24:48.282179 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.282180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.282194 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.282258 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.282891 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.283451 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.284477 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.284689 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.284767 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.289482 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.289483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.289484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.289491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.290975 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.291450 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.291452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.291452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.291519 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.291785 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.291787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.291787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.291923 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.292000 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.292374 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.292469 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:24:48.292780 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.281285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36839 (local address 127.0.0.1:40260)
0929 03:24:48.281466 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.281503 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.282072 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.282328 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.282332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.282593 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.282787 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.282790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.283554 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.283555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.284374 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.284376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.284775 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.284775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.284910 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.285502 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.285510 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.289370 (+ 3860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.291059 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.291066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.291068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.291368 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.291594 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.291595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.291596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.291715 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.292006 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.292014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.292218 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.292336 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.292525 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260929 03:24:48.293355 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:24:48.301993 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.293635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37656)
0929 03:24:48.293683 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.293694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.293734 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.294038 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.294039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.294185 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.294230 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.294231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.295104 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.295105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.295673 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.295675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.296176 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.296177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.296244 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.296636 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.296640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.298783 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.300714 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.300715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.300716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.300969 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.301168 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.301168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.301168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.301245 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.301621 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.301622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.301662 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.301714 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.301802 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:24:48.302007 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.293740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.27.68.125:34911)
0929 03:24:48.293894 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.293899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.293917 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:24:48.293938 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.293944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.293955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.294038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.294304 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.295012 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.295765 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.296085 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.296176 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.298869 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.298878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.298878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.298885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.300623 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.301053 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.301056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.301056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.301111 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.301315 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.301315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.301315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.301558 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.301631 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.301747 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.301897 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260929 03:24:48.302711 8920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.303124 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.303004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60422)
0929 03:24:48.303050 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.303103 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:24:48.303323 8869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
I20260929 03:24:48.303802 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.303711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60438)
0929 03:24:48.303743 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.303772 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 03:24:48.303918 8920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:43219: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
I20260929 03:24:48.304281 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:43219
--webserver_interface=127.27.68.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:48.339308 8845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } attempt: 3
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:24:48.347090 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.339795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37662)
0929 03:24:48.339859 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.339874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.339948 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.340131 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.340132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.340285 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.340354 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.340356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.341302 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.341303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.341835 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.341836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.342311 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.342311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.342368 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.342859 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.342863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.344278 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.346028 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.346028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.346029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.346257 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.346489 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.346490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.346490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.346578 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.346844 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.346844 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.346883 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.346938 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.347016 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:24:48.347149 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.339889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.27.68.125:34911)
0929 03:24:48.339974 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.339977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.339990 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:24:48.340024 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.340025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.340041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.340120 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.340469 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.341219 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.341916 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.342235 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.342306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.344359 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.344360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.344360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.344369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.345954 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.346328 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.346330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.346330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.346388 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.346640 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.346641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.346641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.346772 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.346848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.346977 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.347058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:24:48.347568 8845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.347927 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.347828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60454)
0929 03:24:48.347877 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.347908 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:24:48.348322 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.348234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60470)
0929 03:24:48.348276 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.348300 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:24:48.348413 8845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:43219: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
I20260929 03:24:48.353153 8920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } attempt: 1
I20260929 03:24:48.353627 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.353493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60482)
0929 03:24:48.353556 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.353598 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:24:48.353999 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.353905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60490)
0929 03:24:48.353952 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.353980 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:24:48.354095 8920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:43219: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
I20260929 03:24:48.379555 8845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } attempt: 1
I20260929 03:24:48.380016 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.379882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60492)
0929 03:24:48.379945 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.379987 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:24:48.380430 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.380337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60498)
0929 03:24:48.380386 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.380413 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:24:48.380532 8845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:43219: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
W20260929 03:24:48.388762 8927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:48.389026 8927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:48.389066 8927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:48.390280 8927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:48.390353 8927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:48.390381 8927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:48.390413 8927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:48.390437 8927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:48.392141 8927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:36839,127.27.68.125:34911,127.27.68.124:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.68.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:48.392434 8927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:48.392670 8927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:48.397635 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.397635 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:48.397670 8932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:48.397929 8927 server_base.cc:1034] running on GCE node
I20260929 03:24:48.398227 8927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:48.398556 8927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:48.399746 8927 hybrid_clock.cc:648] HybridClock initialized: now 1790652288399716 us; error 56 us; skew 500 ppm
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:48.402263 8927 init.cc:377] Logged in from keytab as kudu/127.27.68.124@KRBTEST.COM (short username kudu)
I20260929 03:24:48.402716 8927 webserver.cc:460] Webserver started at http://127.27.68.124:37015/ using document root <none> and password file <none>
I20260929 03:24:48.402963 8927 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:48.403035 8927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:48.403194 8927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:48.404312 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data/instance:
uuid: "2277456b11264c18ada3107fb46e8517"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "3af6c4b95af418a156a2e8808f1f2b2c"
server_key_iv: "7110d8b36a30e80a4a0d30b702d82809"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.404690 8927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal/instance:
uuid: "2277456b11264c18ada3107fb46e8517"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "3af6c4b95af418a156a2e8808f1f2b2c"
server_key_iv: "7110d8b36a30e80a4a0d30b702d82809"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.406032 8927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:24:48.406750 8942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:48.406926 8927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:48.407034 8927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "2277456b11264c18ada3107fb46e8517"
format_stamp: "Formatted at 2026-09-29 03:24:48 on dist-test-slave-fzwk"
server_key: "3af6c4b95af418a156a2e8808f1f2b2c"
server_key_iv: "7110d8b36a30e80a4a0d30b702d82809"
server_key_version: "encryptionkey@0"
I20260929 03:24:48.407119 8927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:48.418957 8927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:48.421358 8927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:48.421530 8927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:48.422224 8920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } attempt: 2
I20260929 03:24:48.422616 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.422489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60506)
0929 03:24:48.422549 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.422586 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:24:48.422923 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.422832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60510)
0929 03:24:48.422876 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.422900 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:24:48.423007 8920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:43219: Network error: Client connection negotiation failed: client connection to 127.27.68.124:43219: connect: Connection refused (error 111)
I20260929 03:24:48.425581 8927 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.124:43219
I20260929 03:24:48.425613 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.124:43219 every 8 connection(s)
I20260929 03:24:48.425987 8927 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
I20260929 03:24:48.426885 8995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:48.428476 8995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.428997 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 8927
I20260929 03:24:48.429117 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652132259900-27921-0/minicluster-data/master-2/wal/instance
I20260929 03:24:48.429327 27921 external_mini_cluster.cc:1428] Setting key 10dcee9370de328b7c88c2aaa5350106
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:24:48.439143 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.429213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.27.68.126:36839)
0929 03:24:48.429262 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.429265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.429567 (+ 302us) server_negotiation.cc:407] Connection header received
0929 03:24:48.430232 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.430234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.430246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.430317 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.431075 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.431841 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.433063 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.433350 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.433421 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.435910 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.435912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.435912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.435921 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.437568 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.437969 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.437970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.437971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.438037 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.438372 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.438373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.438373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.438549 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.438616 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.438937 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.439033 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:24:48.439352 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.429200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36839 (local address 127.0.0.1:40262)
0929 03:24:48.429436 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.429488 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.430137 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.430354 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.430357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.430724 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.430980 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.430986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.431922 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.431922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.432969 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.432972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.433432 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.433433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.433542 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.434083 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.434095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.435811 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.437662 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.437668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.437671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.437885 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.438108 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.438109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.438109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.438293 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.438621 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.438628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.438805 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.438888 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.439061 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260929 03:24:48.439352 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.429733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.27.68.124:43219)
0929 03:24:48.429899 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.429904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.429941 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:24:48.430152 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.430162 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.430205 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.430310 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.431062 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.432182 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.433163 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.433607 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.433655 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.436430 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.436441 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.436441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.436449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.438099 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.438478 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.438481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.438482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.438537 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.438750 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.438751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.438751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.438903 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.439072 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.439143 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.439245 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260929 03:24:48.440044 8995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:48.447340 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.440375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.27.68.125:34911)
0929 03:24:48.440419 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.440420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.440428 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:24:48.440457 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.440458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.440469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.440528 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.440834 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.441467 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.442319 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.442503 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.442593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.444559 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.444560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.444560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.444567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.446128 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.446483 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.446484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.446484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.446540 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.446742 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.446743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.446743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.446886 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.446965 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.447137 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.447197 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:24:48.447362 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.440291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:34911 (local address 127.0.0.1:37676)
0929 03:24:48.440341 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.440350 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.440386 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.440546 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.440547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.440697 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.440761 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.440762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.441550 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.441550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.442234 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.442236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.442604 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.442608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.442664 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.443060 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.443065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.444449 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.446218 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.446219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.446220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.446411 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.446615 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.446616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.446616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.446677 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.446963 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.446964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.447029 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.447117 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.447251 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:24:48.447822 8995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:24:48.448276 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.440414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.27.68.126:36839)
0929 03:24:48.440454 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.440456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.440493 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:24:48.440574 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.440574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.440583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.440620 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.441143 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.441728 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.442570 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.442827 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.442889 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.445417 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.445431 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.445439 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.445447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.446935 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.447315 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.447316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.447316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.447370 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.447718 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.447719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.447719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.447846 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.448038 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.448093 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.448177 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:24:48.455050 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.448088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60536)
0929 03:24:48.448131 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.448142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.448176 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.448312 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.448312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.448458 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.448489 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.448490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.449300 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.449301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.449885 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.449887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.450307 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.450307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.450362 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.450740 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.450745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.452181 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.454006 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.454007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.454008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.454226 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.454443 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.454443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.454444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.454495 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.454768 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.454768 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.454802 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.454853 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.454930 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:24:48.455156 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.448145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.27.68.124:43219)
0929 03:24:48.448182 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.448184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.448192 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:24:48.448214 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.448215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.448226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.448279 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.448556 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.449226 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.449961 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.450234 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.450284 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.452272 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.452273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.452273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.452279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.453886 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.454299 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.454301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.454301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.454361 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.454559 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.454560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.454561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.454698 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.454775 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.454880 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.454986 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":19}
I20260929 03:24:48.455669 8845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } attempt: 2
I20260929 03:24:48.456676 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.448875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.27.68.125:34911)
0929 03:24:48.448910 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.448911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.448919 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:24:48.448966 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.448966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.448975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.449033 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.449515 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.449963 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.450802 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.450979 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.451030 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.453257 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.453259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.453259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.453267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.454942 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.455822 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.455824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.455824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.455889 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.456099 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.456100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.456100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.456224 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.456431 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.456470 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.456554 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:24:48.457687 8995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: Bootstrap starting.
I20260929 03:24:48.458653 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:24:48.458860 8995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:48.459187 8995 log.cc:824] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:48.460168 8995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: No bootstrap required, opened a new log
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:24:48.462288 8995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.462419 8995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:48.462442 8995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2277456b11264c18ada3107fb46e8517, State: Initialized, Role: FOLLOWER
I20260929 03:24:48.462595 8995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.462889 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.462947 8995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:48.463076 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:24:48.463819 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.456722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60546)
0929 03:24:48.456775 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.456785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.456824 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.456993 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.456994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.457165 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.457225 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.457227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.458097 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.458098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.458590 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.458591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.459020 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.459021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.459078 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.459583 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.459589 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.461021 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.462525 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.462526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.462526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.462779 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.463005 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.463006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.463007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.463091 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.463456 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.463457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.463507 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.463559 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.463674 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:24:48.464192 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.456803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.27.68.124:43219)
0929 03:24:48.456867 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.456869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.456877 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:24:48.456898 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.456899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.456908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.456962 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.457310 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.457999 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.458687 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.458951 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.459010 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.461116 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.461117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.461117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.461123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.462407 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.462851 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.462852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.462853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.462915 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.463173 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.463174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.463175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.463368 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.463950 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.463996 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.464090 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:24:48.465163 9019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:24:48.465227 9019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:24:48.466296 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: Bootstrap starting.
I20260929 03:24:48.467104 8845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:48.468065 8845 log.cc:824] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:48.471999 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.463713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.27.68.125:34911)
0929 03:24:48.463755 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.463757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.464014 (+ 257us) server_negotiation.cc:407] Connection header received
0929 03:24:48.464107 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.464108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.464120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.464169 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.465317 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.465999 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.466946 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.467355 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.467395 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.468823 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.468824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.468824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.468830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.470566 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.471079 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.471079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.471080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.471140 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.471416 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.471417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.471417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.471540 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.471630 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.471819 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.471894 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:24:48.469473 8927 master_runner.cc:186] Error getting master registration for 127.27.68.126:36839: OK, instance_id { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" instance_seqno: 1790652288110619 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:24:48.472261 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.463470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.27.68.124:43219)
0929 03:24:48.463805 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.463807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.463815 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:24:48.463836 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.463837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.463847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.464662 (+ 815us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.464833 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.465720 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.466760 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.467079 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.467415 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.468729 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.468731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.468731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.468739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.470523 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.471071 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.471072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.471073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.471140 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.471411 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.471411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.471412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.471540 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.472012 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.472068 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.472158 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":67,"threads_started":1}
I20260929 03:24:48.472282 8927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:24:48.476014 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: No bootstrap required, opened a new log
I20260929 03:24:48.478084 8845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.478237 8845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:48.478273 8845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e954352ccd44408c66bb6892aa62b1, State: Initialized, Role: FOLLOWER
I20260929 03:24:48.478394 8845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.478736 8845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:48.478685 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.478893 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:24:48.480615 9033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:24:48.481297 9033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:24:48.482478 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.475765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.27.68.126:36839)
0929 03:24:48.475809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.475811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.475850 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:24:48.475926 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.475927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.475946 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.476019 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.476537 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.477001 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.477657 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.477849 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.477941 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.479419 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.479421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.479421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.479429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.480977 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.481396 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.481397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.481398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.481464 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.481652 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.481653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.481653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.481782 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.481875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.482321 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.482394 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:24:48.482267 8777 master_runner.cc:186] Error getting master registration for 127.27.68.125:34911: OK, instance_id { permanent_uuid: "6629454f0a874005a860583b03dd9f27" instance_seqno: 1790652288275643 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:24:48.484637 8777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:24:48.514133 8920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } attempt: 3
Sep 29 03:24:48 dist-test-slave-fzwk krb5kdc[8764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:24:48.522614 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.514415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:43219 (local address 127.0.0.1:60562)
0929 03:24:48.514476 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:48.514488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:48.514530 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:48.514708 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:48.514709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:48.514907 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:48.514974 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.514976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.515974 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.515976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.516698 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:48.516702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.517277 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.517278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:48.517333 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.517871 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:48.517877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:48.519548 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:48.521513 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.521514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.521515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.521731 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.521930 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:48.521931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:48.521931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:48.521985 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:48.522298 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:48.522299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:48.522345 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:24:48.522407 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:24:48.522506 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:24:48.523301 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:48.514494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.27.68.124:43219)
0929 03:24:48.514560 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:48.514562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:48.514571 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:24:48.514599 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:48.514600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:48.514613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:48.514696 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:48.515074 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.515865 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.516800 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:48.517147 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:48.517219 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:48.519651 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:48.519653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:48.519653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:48.519661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:48.521401 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.521809 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.521811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.521811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.521866 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:48.522060 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:48.522061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:48.522061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:48.522210 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:48.522270 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:48.523081 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 03:24:48.523175 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:24:48.525249 8920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: Bootstrap starting.
I20260929 03:24:48.526013 8920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:48.526351 8920 log.cc:824] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:48.527330 8920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: No bootstrap required, opened a new log
I20260929 03:24:48.529395 8920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.529548 8920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:48.529585 8920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6629454f0a874005a860583b03dd9f27, State: Initialized, Role: FOLLOWER
I20260929 03:24:48.529758 8920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.530122 8920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:48.530078 9036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.530285 9036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:24:48.531991 9047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:24:48.532119 9047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:24:48.534106 8852 master_runner.cc:263] No leader master found from master 6629454f0a874005a860583b03dd9f27
I20260929 03:24:48.534169 8852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:24:48.608450 8798 tablet.cc:2369] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:24:48.672231 8948 tablet.cc:2369] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:24:48.703161 9005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:48.703418 9005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.703760 9005 leader_election.cc:288] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94e954352ccd44408c66bb6892aa62b1 (127.27.68.126:36839), 6629454f0a874005a860583b03dd9f27 (127.27.68.125:34911)
I20260929 03:24:48.704061 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2277456b11264c18ada3107fb46e8517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e954352ccd44408c66bb6892aa62b1" is_pre_election: true
I20260929 03:24:48.704068 8894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2277456b11264c18ada3107fb46e8517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6629454f0a874005a860583b03dd9f27" is_pre_election: true
I20260929 03:24:48.704229 8894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2277456b11264c18ada3107fb46e8517 in term 0.
I20260929 03:24:48.704236 8819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2277456b11264c18ada3107fb46e8517 in term 0.
I20260929 03:24:48.704430 8945 leader_election.cc:302] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2277456b11264c18ada3107fb46e8517, 94e954352ccd44408c66bb6892aa62b1; no voters:
I20260929 03:24:48.704594 9005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:24:48.704670 9005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:24:48.704705 9005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:48.705443 9005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.705601 9005 leader_election.cc:288] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [CANDIDATE]: Term 1 election: Requested vote from peers 94e954352ccd44408c66bb6892aa62b1 (127.27.68.126:36839), 6629454f0a874005a860583b03dd9f27 (127.27.68.125:34911)
I20260929 03:24:48.705780 8819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2277456b11264c18ada3107fb46e8517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e954352ccd44408c66bb6892aa62b1"
I20260929 03:24:48.705806 8894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2277456b11264c18ada3107fb46e8517" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6629454f0a874005a860583b03dd9f27"
I20260929 03:24:48.705904 8819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:48.705925 8894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:48.706648 8894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2277456b11264c18ada3107fb46e8517 in term 1.
I20260929 03:24:48.706656 8819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2277456b11264c18ada3107fb46e8517 in term 1.
I20260929 03:24:48.706830 8945 leader_election.cc:302] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2277456b11264c18ada3107fb46e8517, 94e954352ccd44408c66bb6892aa62b1; no voters:
I20260929 03:24:48.706981 9005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:48.707173 9005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [term 1 LEADER]: Becoming Leader. State: Replica: 2277456b11264c18ada3107fb46e8517, State: Running, Role: LEADER
I20260929 03:24:48.707295 9005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } }
I20260929 03:24:48.707712 9049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2277456b11264c18ada3107fb46e8517. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.707805 9049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:48.707995 9051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:48.708187 9051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:48.709486 8894 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [term 1 FOLLOWER]: Refusing update from remote peer 2277456b11264c18ada3107fb46e8517: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:48.709748 8819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [term 1 FOLLOWER]: Refusing update from remote peer 2277456b11264c18ada3107fb46e8517: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:48.709828 9005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:48.710088 9005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:48.711052 9036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2277456b11264c18ada3107fb46e8517. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.711146 9036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:24:48.711994 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.712112 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:48.712301 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2277456b11264c18ada3107fb46e8517. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.712400 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:24:48.712457 9036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.712525 9036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:24:48.713008 9053 mvcc.cc:205] Tried to move back new op lower bound from 7334511774551805952 to 7334511774545678336. Current Snapshot: MvccSnapshot[applied={T|T < 7334511774551805952}]
I20260929 03:24:48.713258 9049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.713352 9049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517 [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:48.713443 9051 catalog_manager.cc:1347] Generated new cluster ID: 6c201b66388040e8bd3cc6e24de31e16
I20260929 03:24:48.713485 9051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:24:48.713735 9022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2277456b11264c18ada3107fb46e8517" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e954352ccd44408c66bb6892aa62b1" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36839 } } peers { permanent_uuid: "6629454f0a874005a860583b03dd9f27" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 34911 } } peers { permanent_uuid: "2277456b11264c18ada3107fb46e8517" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 43219 } } }
I20260929 03:24:48.713807 9022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:24:48.726809 9051 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:24:48.727542 9051 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:48.732280 9051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2277456b11264c18ada3107fb46e8517: Generated new TSK 0
I20260929 03:24:48.732519 9051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:24:49.481753 9033 catalog_manager.cc:1259] Loaded cluster ID: 6c201b66388040e8bd3cc6e24de31e16
I20260929 03:24:49.481846 9033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: loading cluster ID for follower catalog manager: success
I20260929 03:24:49.482662 9033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: acquiring CA information for follower catalog manager: success
I20260929 03:24:49.482971 9033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94e954352ccd44408c66bb6892aa62b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:24:49.532585 9047 catalog_manager.cc:1259] Loaded cluster ID: 6c201b66388040e8bd3cc6e24de31e16
I20260929 03:24:49.532665 9047 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: loading cluster ID for follower catalog manager: success
I20260929 03:24:49.533428 9047 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: acquiring CA information for follower catalog manager: success
I20260929 03:24:49.533671 9047 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6629454f0a874005a860583b03dd9f27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:24:49.799392 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8777
I20260929 03:24:49.804934 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8852
I20260929 03:24:49.810045 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 8927
2026-09-29T03:24:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2965 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:24:49 dist-test-slave-fzwk krb5kdc[9067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:24:49 dist-test-slave-fzwk krb5kdc[9067](info): set up 2 sockets
Sep 29 03:24:49 dist-test-slave-fzwk krb5kdc[9067](info): commencing operation
krb5kdc: starting...
W20260929 03:24:50.843654 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:24:50 dist-test-slave-fzwk krb5kdc[9067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:24:50.964424 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41299
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:51.047843 9080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:51.048075 9080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:51.048099 9080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:51.049235 9080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:51.049273 9080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:51.049286 9080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:51.049299 9080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:51.049311 9080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:51.050911 9080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:41299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:51.051131 9080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:51.051353 9080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:51.056289 9085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:51.056293 9086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:51.056536 9080 server_base.cc:1034] running on GCE node
W20260929 03:24:51.056321 9088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:51.056806 9080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:51.057156 9080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:51.058336 9080 hybrid_clock.cc:648] HybridClock initialized: now 1790652291058323 us; error 30 us; skew 500 ppm
Sep 29 03:24:51 dist-test-slave-fzwk krb5kdc[9067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652291, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:51.060968 9080 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:51.061457 9080 webserver.cc:460] Webserver started at http://127.27.68.126:45079/ using document root <none> and password file <none>
I20260929 03:24:51.061713 9080 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:51.061784 9080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:51.061936 9080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:51.063040 9080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "d8da24ec014f41648885ff605a2df513"
format_stamp: "Formatted at 2026-09-29 03:24:51 on dist-test-slave-fzwk"
server_key: "8a19f0f646dc8eef88f5d704b2b8df5e"
server_key_iv: "6aa0f628572c4df00b37adfd5971ceab"
server_key_version: "encryptionkey@0"
I20260929 03:24:51.063445 9080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "d8da24ec014f41648885ff605a2df513"
format_stamp: "Formatted at 2026-09-29 03:24:51 on dist-test-slave-fzwk"
server_key: "8a19f0f646dc8eef88f5d704b2b8df5e"
server_key_iv: "6aa0f628572c4df00b37adfd5971ceab"
server_key_version: "encryptionkey@0"
I20260929 03:24:51.064818 9080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:24:51.065579 9095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:51.065757 9080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:24:51.065867 9080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "d8da24ec014f41648885ff605a2df513"
format_stamp: "Formatted at 2026-09-29 03:24:51 on dist-test-slave-fzwk"
server_key: "8a19f0f646dc8eef88f5d704b2b8df5e"
server_key_iv: "6aa0f628572c4df00b37adfd5971ceab"
server_key_version: "encryptionkey@0"
I20260929 03:24:51.065949 9080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:51.079993 9080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:51.082517 9080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:51.082706 9080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:51.086812 9080 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:41299
I20260929 03:24:51.086846 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:41299 every 8 connection(s)
I20260929 03:24:51.087229 9080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:51.088135 9148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:51.090389 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: Bootstrap starting.
I20260929 03:24:51.090979 9148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:51.091061 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9080
I20260929 03:24:51.091185 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:24:51.091356 9148 log.cc:824] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:51.091409 27921 external_mini_cluster.cc:1428] Setting key a033dadc6cf6a4c5a2dffd2e9892f574
I20260929 03:24:51.092209 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: No bootstrap required, opened a new log
I20260929 03:24:51.094383 9148 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } }
I20260929 03:24:51.094614 9148 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:51.094678 9148 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da24ec014f41648885ff605a2df513, State: Initialized, Role: FOLLOWER
I20260929 03:24:51.094825 9148 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } }
I20260929 03:24:51.094928 9148 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:51.094998 9148 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:51.095059 9148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:51.095947 9148 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } }
I20260929 03:24:51.096082 9148 leader_election.cc:302] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8da24ec014f41648885ff605a2df513; no voters:
I20260929 03:24:51.096293 9148 leader_election.cc:288] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:24:51.096346 9153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:51.096530 9153 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [term 1 LEADER]: Becoming Leader. State: Replica: d8da24ec014f41648885ff605a2df513, State: Running, Role: LEADER
I20260929 03:24:51.096577 9148 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:51.096657 9153 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } }
I20260929 03:24:51.097085 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8da24ec014f41648885ff605a2df513. Latest consensus state: current_term: 1 leader_uuid: "d8da24ec014f41648885ff605a2df513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } } }
I20260929 03:24:51.097100 9154 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8da24ec014f41648885ff605a2df513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8da24ec014f41648885ff605a2df513" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41299 } } }
I20260929 03:24:51.097229 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:51.098244 9154 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513 [sys.catalog]: This master's current role is: LEADER
W20260929 03:24:51.098459 9168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:24:51.098788 9168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:24:51.098896 9170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:51.099102 9170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:24:51 dist-test-slave-fzwk krb5kdc[9067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:51.101202 9170 catalog_manager.cc:1347] Generated new cluster ID: e414565e6a24453f839f5bf2c929c728
I20260929 03:24:51.101265 9170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:24:51.103617 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:51.091961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.27.68.126:41299)
0929 03:24:51.092167 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:51.092178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:51.092213 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:24:51.092708 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:51.092716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:51.092766 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:51.092916 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:51.093566 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:51.094480 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:51.095903 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:51.096549 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:51.096691 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:51.099848 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:51.099853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:51.099853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:51.099867 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:51.101916 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:51.102333 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:51.102336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:51.102337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:51.102439 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:51.102661 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:51.102662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:51.102662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:51.102928 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:51.103128 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:51.103199 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:24:51.103389 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260929 03:24:51.104843 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:24:51.129930 9170 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:24:51.130548 9170 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:51.136181 9170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8da24ec014f41648885ff605a2df513: Generated new TSK 0
I20260929 03:24:51.136431 9170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:24:51.158838 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9080
2026-09-29T03:24:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:24:51 dist-test-slave-fzwk krb5kdc[9175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:24:51 dist-test-slave-fzwk krb5kdc[9175](info): set up 2 sockets
Sep 29 03:24:51 dist-test-slave-fzwk krb5kdc[9175](info): commencing operation
krb5kdc: starting...
W20260929 03:24:52.187682 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:24:52.306559 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:52.389930 9188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:52.390192 9188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:52.390230 9188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:52.391489 9188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:52.391567 9188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:52.391592 9188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:52.391613 9188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:52.391642 9188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:52.393242 9188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:52.393555 9188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:52.393800 9188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:52.398836 9193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.398844 9194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.398825 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:52.399036 9188 server_base.cc:1034] running on GCE node
I20260929 03:24:52.399395 9188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:52.399714 9188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:52.400889 9188 hybrid_clock.cc:648] HybridClock initialized: now 1790652292400876 us; error 33 us; skew 500 ppm
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:52.403522 9188 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:52.404027 9188 webserver.cc:460] Webserver started at http://127.27.68.126:33623/ using document root <none> and password file <none>
I20260929 03:24:52.404305 9188 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:52.404395 9188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:52.404556 9188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:52.405689 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "fcfb7488032e4feab656f559b806741e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "2f7edd95289f167ed31a717906efe6f2"
server_key_iv: "b0735d424d0a12cc1c4089d3c6d58611"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.406075 9188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "fcfb7488032e4feab656f559b806741e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "2f7edd95289f167ed31a717906efe6f2"
server_key_iv: "b0735d424d0a12cc1c4089d3c6d58611"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.407546 9188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 03:24:52.408334 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.408514 9188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:52.408629 9188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fcfb7488032e4feab656f559b806741e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "2f7edd95289f167ed31a717906efe6f2"
server_key_iv: "b0735d424d0a12cc1c4089d3c6d58611"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.408710 9188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:52.416224 9188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:52.418934 9188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:52.419116 9188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:52.423285 9188 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44107
I20260929 03:24:52.423317 9255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44107 every 8 connection(s)
I20260929 03:24:52.423691 9188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:52.424693 9256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:52.427017 9256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap starting.
I20260929 03:24:52.427659 9256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:52.427929 9256 log.cc:824] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:52.428613 9256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: No bootstrap required, opened a new log
I20260929 03:24:52.430186 9256 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:52.430333 9256 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:52.430356 9256 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfb7488032e4feab656f559b806741e, State: Initialized, Role: FOLLOWER
I20260929 03:24:52.430462 9256 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:52.430517 9256 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:52.430538 9256 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:52.430565 9256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:52.431428 9256 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:52.431558 9256 leader_election.cc:302] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfb7488032e4feab656f559b806741e; no voters:
I20260929 03:24:52.431725 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9188
I20260929 03:24:52.431747 9256 leader_election.cc:288] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:24:52.431805 9259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:52.431842 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:24:52.431969 9259 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 LEADER]: Becoming Leader. State: Replica: fcfb7488032e4feab656f559b806741e, State: Running, Role: LEADER
I20260929 03:24:52.432027 27921 external_mini_cluster.cc:1428] Setting key 0554f7bf02b53c54f9305b532cc5ccd8
I20260929 03:24:52.432041 9256 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:52.432114 9259 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:52.432476 9261 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfb7488032e4feab656f559b806741e. Latest consensus state: current_term: 1 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:52.432551 9261 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:52.432705 9260 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:52.432770 9260 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:52.432943 9266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:52.433512 9266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:52.435318 9266 catalog_manager.cc:1347] Generated new cluster ID: 68257998dfc446368cda8ee42c9e4d5d
I20260929 03:24:52.435425 9266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:52.442711 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.432683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.27.68.126:44107)
0929 03:24:52.432853 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.432864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.432915 (+ 51us) server_negotiation.cc:407] Connection header received
0929 03:24:52.433044 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.433052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.433118 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.434091 (+ 973us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:52.434221 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.435490 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.436399 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.436832 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.436960 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.439422 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:52.439431 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:52.439432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:52.439444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:52.441158 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.441533 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.441536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.441536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.441643 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.441852 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.441853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.441853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.442103 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:52.442294 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:52.442359 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:24:52.442507 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260929 03:24:52.465816 9266 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:24:52.466585 9266 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:52.471078 9266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Generated new TSK 0
I20260929 03:24:52.471320 9266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:24:52.497906 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:52.584956 9282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:52.585161 9282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:52.585186 9282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:52.586359 9282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:52.586406 9282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:52.586447 9282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:24:52.587966 9282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:52.588219 9282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:52.588482 9282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:52.593683 9288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.593647 9290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:52.593663 9282 server_base.cc:1034] running on GCE node
W20260929 03:24:52.593647 9287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:52.594121 9282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:52.594380 9282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:52.595554 9282 hybrid_clock.cc:648] HybridClock initialized: now 1790652292595541 us; error 43 us; skew 500 ppm
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:52.598137 9282 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:24:52.598542 9282 webserver.cc:460] Webserver started at http://127.27.68.65:46227/ using document root <none> and password file <none>
I20260929 03:24:52.598737 9282 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:52.598778 9282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:52.598866 9282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:52.599934 9282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "e68dd44f42ee4e39ab7ac5408a42c394"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "382ad5d12306e0b1a43d5efdcaedad59"
server_key_iv: "2884ca8cf7a3f226a64dec30fe5d2a1c"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.600224 9282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "e68dd44f42ee4e39ab7ac5408a42c394"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "382ad5d12306e0b1a43d5efdcaedad59"
server_key_iv: "2884ca8cf7a3f226a64dec30fe5d2a1c"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.601567 9282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:24:52.602254 9297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.602449 9282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:52.602567 9282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "e68dd44f42ee4e39ab7ac5408a42c394"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "382ad5d12306e0b1a43d5efdcaedad59"
server_key_iv: "2884ca8cf7a3f226a64dec30fe5d2a1c"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.602643 9282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:52.624183 9282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:52.626652 9282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:52.626776 9282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:52.626971 9282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:52.627385 9282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:52.627460 9282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.627504 9282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:52.627532 9282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.633692 9282 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:34335
I20260929 03:24:52.633777 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:34335 every 8 connection(s)
I20260929 03:24:52.634123 9282 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:52.643265 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9282
I20260929 03:24:52.643376 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:24:52.643606 27921 external_mini_cluster.cc:1428] Setting key 1200fffb092cca9b8e1774d7e0c78773
I20260929 03:24:52.643889 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.634963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:58949 (local address 127.27.68.126:44107)
0929 03:24:52.635065 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.635068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.635275 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:24:52.635835 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.635837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.635853 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.635911 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:52.636916 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.637436 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.638481 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.638708 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.638796 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.641062 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:52.641064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:52.641064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:52.641072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:52.642348 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.642753 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.642754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.642755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.642808 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.643031 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.643032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.643032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.643159 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:52.643262 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:52.643631 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 03:24:52.643743 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:24:52.644066 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:58949)
0929 03:24:52.635115 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.635165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.635725 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.635987 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.635991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.636600 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:52.636807 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.636812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.637548 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.637550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.638382 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.638385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.638812 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.638813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.638952 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.639520 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:52.639529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:52.640931 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:52.642442 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.642449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.642451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.642682 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.642869 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.642870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.642870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.642978 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.643271 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:52.643279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:52.643478 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 03:24:52.643610 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 03:24:52.643797 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260929 03:24:52.645828 9411 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:52.645967 9411 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:52.646204 9411 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:52.646736 9220 ts_manager.cc:194] Registered new tserver with Master: e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65:34335)
I20260929 03:24:52.647449 9220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:58949
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:24:52.669966 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:52.751653 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:52.751883 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:52.751907 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:52.753132 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:52.753185 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:52.753211 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:24:52.754703 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:52.754966 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:52.755239 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:52.760315 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.760313 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.760300 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:52.760641 9416 server_base.cc:1034] running on GCE node
I20260929 03:24:52.760879 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:52.761134 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:52.762320 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790652292762300 us; error 33 us; skew 500 ppm
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:52.764875 9416 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:24:52.765383 9416 webserver.cc:460] Webserver started at http://127.27.68.66:39857/ using document root <none> and password file <none>
I20260929 03:24:52.765625 9416 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:52.765693 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:52.765844 9416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:52.766912 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "5d091f993b0245298709088576cb2ae6"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "f6797c56a420a13636cd380cb32459f5"
server_key_iv: "2b56991f295f679561f6b7a56bc64a37"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.767359 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "5d091f993b0245298709088576cb2ae6"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "f6797c56a420a13636cd380cb32459f5"
server_key_iv: "2b56991f295f679561f6b7a56bc64a37"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.768744 9416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:52.769521 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.769718 9416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:52.769814 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "5d091f993b0245298709088576cb2ae6"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "f6797c56a420a13636cd380cb32459f5"
server_key_iv: "2b56991f295f679561f6b7a56bc64a37"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.769901 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:52.784847 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:52.786783 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:52.786953 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:52.787201 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:52.787590 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:52.787643 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.787688 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:52.787714 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.793713 9416 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:34261
I20260929 03:24:52.793757 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:34261 every 8 connection(s)
I20260929 03:24:52.794214 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:24:52.794853 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9416
I20260929 03:24:52.794976 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:24:52.795197 27921 external_mini_cluster.cc:1428] Setting key dc53567c8e0a8b1c1ce71226990e73df
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
I20260929 03:24:52.805511 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.795395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:38735 (local address 127.27.68.126:44107)
0929 03:24:52.795446 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.795448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.795630 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:24:52.796328 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.796330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.796341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.796398 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:52.797796 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.798607 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.799768 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.799990 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.800074 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.802315 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:52.802318 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:52.802318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:52.802327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:52.804063 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.804471 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.804473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.804474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.804548 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.804796 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.804797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.804798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.804961 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:52.805044 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:52.805289 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:24:52.805384 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:24:52.805716 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.795279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:38735)
0929 03:24:52.795472 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.795543 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.796169 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.796474 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.796478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.797364 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:52.797672 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.797681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.798704 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.798705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.799675 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.799677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.800085 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.800086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.800222 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.800824 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:52.800833 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:52.802210 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:52.804149 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.804156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.804157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.804392 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.804622 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.804623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.804623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.804726 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.805036 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:52.805040 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:52.805161 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 03:24:52.805244 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:24:52.805454 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260929 03:24:52.806195 9545 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:52.806334 9545 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:52.806576 9545 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:52.807000 9220 ts_manager.cc:194] Registered new tserver with Master: 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
I20260929 03:24:52.807471 9220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:38735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:24:52.821906 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:52.916769 9550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:52.916990 9550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:52.917017 9550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:52.918311 9550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:52.918365 9550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:52.918394 9550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:24:52.920107 9550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:52.920408 9550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:52.920662 9550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:52.925787 9556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.925787 9558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:52.925804 9555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:52.926172 9550 server_base.cc:1034] running on GCE node
I20260929 03:24:52.926381 9550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:52.926635 9550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:52.927798 9550 hybrid_clock.cc:648] HybridClock initialized: now 1790652292927781 us; error 35 us; skew 500 ppm
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:52.930344 9550 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:24:52.930842 9550 webserver.cc:460] Webserver started at http://127.27.68.67:44157/ using document root <none> and password file <none>
I20260929 03:24:52.931139 9550 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:52.931217 9550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:52.931399 9550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:24:52.932665 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "a5525e1168764494afa4143a6d73632e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "98b1c613cfd4c9e1d120bc1e1e32d8c4"
server_key_iv: "6667bd40942d271f2ab30aa940cefea1"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.933106 9550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "a5525e1168764494afa4143a6d73632e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "98b1c613cfd4c9e1d120bc1e1e32d8c4"
server_key_iv: "6667bd40942d271f2ab30aa940cefea1"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.934499 9550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:24:52.935362 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.935537 9550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:24:52.935644 9550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "a5525e1168764494afa4143a6d73632e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "98b1c613cfd4c9e1d120bc1e1e32d8c4"
server_key_iv: "6667bd40942d271f2ab30aa940cefea1"
server_key_version: "encryptionkey@0"
I20260929 03:24:52.935731 9550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:52.947914 9550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:52.950428 9550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:52.950624 9550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:52.950881 9550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:52.951272 9550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:52.951330 9550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.951375 9550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:52.951402 9550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:52.957666 9550 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:45743
I20260929 03:24:52.957712 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:45743 every 8 connection(s)
I20260929 03:24:52.958057 9550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:24:52 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652292, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:52.967182 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.958741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:60459 (local address 127.27.68.126:44107)
0929 03:24:52.958831 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.958834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.959053 (+ 219us) server_negotiation.cc:407] Connection header received
0929 03:24:52.959624 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.959625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.959635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.959688 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:52.960654 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.961163 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.962150 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.962330 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.962381 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.964645 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:52.964647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:52.964647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:52.964654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:52.965918 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.966359 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.966360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.966360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.966413 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.966639 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.966639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.966639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.966741 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:52.966791 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:52.967045 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 03:24:52.967112 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:24:52.967331 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9550
I20260929 03:24:52.967406 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:24:52.967507 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.958721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:60459)
0929 03:24:52.958916 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.958957 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.959522 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.959748 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.959752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.960337 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:52.960549 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.960554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.961249 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.961250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.962070 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.962072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.962405 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.962405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:52.962507 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.963091 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:52.963100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:52.964552 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:52.966007 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.966014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.966016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.966266 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.966472 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:52.966473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:52.966473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:52.966585 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:52.966796 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:52.966800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:52.966931 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:24:52.967017 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:24:52.967204 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20260929 03:24:52.967620 27921 external_mini_cluster.cc:1428] Setting key b29bec39e5fee3cbfb0a96343418f2ee
I20260929 03:24:52.968165 9679 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:52.968276 9679 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:52.968474 9679 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:52.968856 9219 ts_manager.cc:194] Registered new tserver with Master: a5525e1168764494afa4143a6d73632e (127.27.68.67:45743)
I20260929 03:24:52.969264 9219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:60459
I20260929 03:24:52.982995 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:24:52.991462 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.984745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.27.68.126:44107)
0929 03:24:52.984832 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.984838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.984930 (+ 92us) server_negotiation.cc:407] Connection header received
0929 03:24:52.985055 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.985056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.985069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.985159 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:52.985825 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.986343 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.987146 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.987427 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.987524 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:52.988852 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:52.988854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:52.988854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:52.988863 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:52.990140 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.990578 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.990579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.990580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.990644 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:52.990836 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:52.990837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:52.990837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:52.990998 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:52.991078 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:52.991195 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:24:52.991341 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:24:52.993235 9219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:24:52.994793 9219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:24:53.001482 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:45743 (local address 127.0.0.1:43446)
0929 03:24:52.997632 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.997641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.997676 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.998119 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.998120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.998122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:52.998177 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.998179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.999514 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.999515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.000398 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.000401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.001108 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.001196 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001219 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.001328 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.001388 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260929 03:24:53.001479 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.27.68.67:45743)
0929 03:24:52.997663 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.997668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.997680 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:24:52.997779 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.997782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.997916 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.997997 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:52.998263 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.999403 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.000486 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.000997 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001104 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001171 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.001345 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.001388 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260929 03:24:53.001626 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:34261 (local address 127.0.0.1:37588)
0929 03:24:52.997398 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.997442 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.997514 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.997843 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.997847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.997849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:52.997932 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.997934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.999330 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.999331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.000409 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.000413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.001379 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.001447 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001452 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.001483 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.001535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260929 03:24:53.001859 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.27.68.65:34335)
0929 03:24:52.997752 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.997755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.997761 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:24:52.997816 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.997818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.997835 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.998031 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:52.998221 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.998934 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.000518 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.001259 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001332 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001390 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.001725 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.001773 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260929 03:24:53.001879 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.27.68.66:34261)
0929 03:24:52.997664 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:52.997668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:52.997679 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:24:52.997719 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:52.997721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:52.997738 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:52.997829 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:52.998025 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.999185 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.000518 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.001259 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001671 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001741 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.001765 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.001807 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
I20260929 03:24:53.001626 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:52.997240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:34335 (local address 127.0.0.1:54982)
0929 03:24:52.997508 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:52.997518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:52.997560 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:52.998111 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:52.998112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:52.998114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:52.998152 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:52.998154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:52.999040 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:52.999041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.000411 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.000413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.001418 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.001419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.001481 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.001484 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.001529 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.001558 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260929 03:24:53.002694 9613 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 (DEFAULT_TABLE table=test-table [id=e970a790411542b1bbe38ea509210970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.002790 9479 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 (DEFAULT_TABLE table=test-table [id=e970a790411542b1bbe38ea509210970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.003077 9613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b8630004d40748ee48a9ae9ce2431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.003149 9479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b8630004d40748ee48a9ae9ce2431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.003885 9345 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 (DEFAULT_TABLE table=test-table [id=e970a790411542b1bbe38ea509210970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.004212 9345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b8630004d40748ee48a9ae9ce2431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.006176 9696 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Bootstrap starting.
I20260929 03:24:53.006592 9697 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Bootstrap starting.
I20260929 03:24:53.006903 9696 tablet_bootstrap.cc:654] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.007324 9697 tablet_bootstrap.cc:654] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.007616 9696 log.cc:824] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.007930 9697 log.cc:824] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.007920 9698 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap starting.
I20260929 03:24:53.008556 9696 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: No bootstrap required, opened a new log
I20260929 03:24:53.008572 9698 tablet_bootstrap.cc:654] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.008682 9696 ts_tablet_manager.cc:1397] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:24:53.008733 9697 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: No bootstrap required, opened a new log
I20260929 03:24:53.008844 9697 ts_tablet_manager.cc:1397] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:24:53.009130 9698 log.cc:824] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.009914 9698 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: No bootstrap required, opened a new log
I20260929 03:24:53.009985 9698 ts_tablet_manager.cc:1397] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:24:53.010535 9697 raft_consensus.cc:348] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.010732 9697 raft_consensus.cc:374] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.010797 9697 raft_consensus.cc:729] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5525e1168764494afa4143a6d73632e, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.010941 9697 consensus_queue.cc:260] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.010900 9696 raft_consensus.cc:348] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.011267 9697 ts_tablet_manager.cc:1428] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:24:53.011332 9696 raft_consensus.cc:374] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.011448 9679 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
I20260929 03:24:53.011464 9696 raft_consensus.cc:729] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d091f993b0245298709088576cb2ae6, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.011595 9696 consensus_queue.cc:260] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.011896 9545 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
I20260929 03:24:53.011899 9696 ts_tablet_manager.cc:1428] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:24:53.012485 9698 raft_consensus.cc:348] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.012631 9698 raft_consensus.cc:374] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.012696 9698 raft_consensus.cc:729] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.012841 9698 consensus_queue.cc:260] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.013109 9698 ts_tablet_manager.cc:1428] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:24:53.013228 9411 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
W20260929 03:24:53.044955 9546 tablet.cc:2369] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:24:53.052335 9704 raft_consensus.cc:482] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:53.052496 9704 raft_consensus.cc:504] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.052914 9704 leader_election.cc:288] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261), a5525e1168764494afa4143a6d73632e (127.27.68.67:45743)
I20260929 03:24:53.057011 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.053288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:49553 (local address 127.27.68.67:45743)
0929 03:24:53.053367 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.053368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.053377 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:24:53.053408 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.053409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.053420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.053485 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.053705 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.054469 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056036 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.056629 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056711 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.056720 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.056897 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.056940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:24:53.057011 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.053276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:59723 (local address 127.27.68.66:34261)
0929 03:24:53.053324 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.053327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.053374 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:24:53.053434 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.053435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.053445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.053499 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.053706 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.054464 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056036 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.056629 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056712 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.056720 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.056897 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.056944 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:24:53.057032 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.053109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:45743 (local address 127.27.68.65:49553)
0929 03:24:53.053253 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.053277 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.053332 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.053519 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.053520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.053522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.053627 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.053629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.054560 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.054561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.055927 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.055931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.056728 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.056815 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.056817 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.056875 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.056930 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":29}
I20260929 03:24:53.057049 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.053118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:34261 (local address 127.27.68.65:59723)
0929 03:24:53.053323 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.053332 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.053381 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.053519 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.053520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.053522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.053627 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.053629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.054560 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.054561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.055927 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.055931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.056728 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.056729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.056811 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.056817 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.056875 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.056928 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":124,"thread_start_us":124,"threads_started":1}
I20260929 03:24:53.057474 9633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b8630004d40748ee48a9ae9ce2431" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5525e1168764494afa4143a6d73632e" is_pre_election: true
I20260929 03:24:53.057668 9633 raft_consensus.cc:2454] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 0.
I20260929 03:24:53.057610 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b8630004d40748ee48a9ae9ce2431" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d091f993b0245298709088576cb2ae6" is_pre_election: true
I20260929 03:24:53.057773 9499 raft_consensus.cc:2454] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 0.
I20260929 03:24:53.057942 9300 leader_election.cc:302] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5525e1168764494afa4143a6d73632e, e68dd44f42ee4e39ab7ac5408a42c394; no voters:
I20260929 03:24:53.058112 9704 raft_consensus.cc:2781] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:24:53.058173 9704 raft_consensus.cc:482] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:24:53.058236 9704 raft_consensus.cc:3037] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.058971 9704 raft_consensus.cc:504] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.059135 9704 leader_election.cc:288] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 election: Requested vote from peers 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261), a5525e1168764494afa4143a6d73632e (127.27.68.67:45743)
I20260929 03:24:53.059329 9499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b8630004d40748ee48a9ae9ce2431" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d091f993b0245298709088576cb2ae6"
I20260929 03:24:53.059425 9499 raft_consensus.cc:3037] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.059521 9633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b8630004d40748ee48a9ae9ce2431" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5525e1168764494afa4143a6d73632e"
I20260929 03:24:53.059638 9633 raft_consensus.cc:3037] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.060225 9499 raft_consensus.cc:2454] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 1.
I20260929 03:24:53.060386 9633 raft_consensus.cc:2454] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 1.
I20260929 03:24:53.060400 9298 leader_election.cc:302] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d091f993b0245298709088576cb2ae6, e68dd44f42ee4e39ab7ac5408a42c394; no voters:
I20260929 03:24:53.060482 9704 raft_consensus.cc:2781] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:53.060672 9704 raft_consensus.cc:686] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 LEADER]: Becoming Leader. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Running, Role: LEADER
I20260929 03:24:53.060781 9704 consensus_queue.cc:237] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.061683 9220 catalog_manager.cc:5495] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 reported cstate change: term changed from 0 to 1, leader changed from <none> to e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } health_report { overall_health: UNKNOWN } } }
I20260929 03:24:53.077437 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.074945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.27.68.65:34335)
0929 03:24:53.075008 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.075011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.075020 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:24:53.075067 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.075067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.075087 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.075150 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:24:53.075296 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.075761 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.076627 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.076872 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.076930 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.077067 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:24:53.077173 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:24:53.077234 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.077282 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.077299 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:24:53.079382 9499 raft_consensus.cc:1261] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Refusing update from remote peer e68dd44f42ee4e39ab7ac5408a42c394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:53.079380 9633 raft_consensus.cc:1261] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Refusing update from remote peer e68dd44f42ee4e39ab7ac5408a42c394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:53.079787 9706 consensus_queue.cc:1035] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:53.079926 9704 consensus_queue.cc:1035] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:53.082249 9709 mvcc.cc:205] Tried to move back new op lower bound from 7334511792450088960 to 7334511792377536512. Current Snapshot: MvccSnapshot[applied={T|T < 7334511792450088960}]
I20260929 03:24:53.082515 9711 mvcc.cc:205] Tried to move back new op lower bound from 7334511792450088960 to 7334511792377536512. Current Snapshot: MvccSnapshot[applied={T|T < 7334511792450088960}]
I20260929 03:24:53.082975 9710 mvcc.cc:205] Tried to move back new op lower bound from 7334511792450088960 to 7334511792377536512. Current Snapshot: MvccSnapshot[applied={T|T < 7334511792450088960}]
I20260929 03:24:53.084761 9219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:53.084892 9219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:53.085739 9219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Sending DeleteTablet for 3 replicas of tablet bc0b8630004d40748ee48a9ae9ce2431
I20260929 03:24:53.086088 9479 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:37588
I20260929 03:24:53.086134 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9282
I20260929 03:24:53.086117 9345 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:54982
I20260929 03:24:53.086747 9613 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:43446
I20260929 03:24:53.086921 9719 tablet_replica.cc:330] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: stopping tablet replica
I20260929 03:24:53.087078 9719 raft_consensus.cc:2229] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:53.087212 9719 raft_consensus.cc:2258] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:53.087275 9720 tablet_replica.cc:330] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: stopping tablet replica
I20260929 03:24:53.087504 9720 raft_consensus.cc:2229] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:53.087630 9720 raft_consensus.cc:2258] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:53.087838 9719 ts_tablet_manager.cc:1905] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:53.088016 9720 ts_tablet_manager.cc:1905] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:53.089553 9720 ts_tablet_manager.cc:1918] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:53.089654 9720 log.cc:1196] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/bc0b8630004d40748ee48a9ae9ce2431
I20260929 03:24:53.089890 9719 ts_tablet_manager.cc:1918] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:53.089929 9720 ts_tablet_manager.cc:1939] T bc0b8630004d40748ee48a9ae9ce2431 P a5525e1168764494afa4143a6d73632e: Deleting consensus metadata
I20260929 03:24:53.089974 9719 log.cc:1196] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/bc0b8630004d40748ee48a9ae9ce2431
I20260929 03:24:53.090267 9719 ts_tablet_manager.cc:1939] T bc0b8630004d40748ee48a9ae9ce2431 P 5d091f993b0245298709088576cb2ae6: Deleting consensus metadata
I20260929 03:24:53.090443 9206 catalog_manager.cc:4841] TS a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): tablet bc0b8630004d40748ee48a9ae9ce2431 (table test-table [id=e970a790411542b1bbe38ea509210970]) successfully deleted
I20260929 03:24:53.090687 9204 catalog_manager.cc:4841] TS 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): tablet bc0b8630004d40748ee48a9ae9ce2431 (table test-table [id=e970a790411542b1bbe38ea509210970]) successfully deleted
W20260929 03:24:53.093488 9205 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:34335 (error 108)
I20260929 03:24:53.093673 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9416
I20260929 03:24:53.094540 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.094401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:34335 (local address 127.0.0.1:54992)
0929 03:24:53.094462 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.094502 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:24:53.094668 9205 catalog_manager.cc:4584] TS e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65:34335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc0b8630004d40748ee48a9ae9ce2431: Network error: Client connection negotiation failed: client connection to 127.27.68.65:34335: connect: Connection refused (error 111)
I20260929 03:24:53.099120 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9550
I20260929 03:24:53.103449 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9188
I20260929 03:24:53.108968 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--webserver_interface=127.27.68.126
--webserver_port=33623
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:53.191846 9722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:53.192101 9722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:53.192144 9722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:53.193351 9722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:53.193420 9722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:53.193447 9722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:53.193480 9722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:53.193504 9722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:53.195026 9722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=33623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:53.195341 9722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:53.195583 9722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:53.204216 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.204265 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.204255 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.204393 9722 server_base.cc:1034] running on GCE node
I20260929 03:24:53.204725 9722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:53.205065 9722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:53.206235 9722 hybrid_clock.cc:648] HybridClock initialized: now 1790652293206222 us; error 45 us; skew 500 ppm
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:53.213444 9722 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:53.213985 9722 webserver.cc:460] Webserver started at http://127.27.68.126:33623/ using document root <none> and password file <none>
I20260929 03:24:53.214258 9722 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:53.214359 9722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:53.215839 9722 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:24:53.216502 9738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.216678 9722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:53.216776 9722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fcfb7488032e4feab656f559b806741e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "2f7edd95289f167ed31a717906efe6f2"
server_key_iv: "b0735d424d0a12cc1c4089d3c6d58611"
server_key_version: "encryptionkey@0"
I20260929 03:24:53.217202 9722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:53.231842 9722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:53.233740 9722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:53.233917 9722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:53.238171 9722 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44107
I20260929 03:24:53.238204 9790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44107 every 8 connection(s)
I20260929 03:24:53.238615 9722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:53.240271 9791 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:24:53.240960 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap starting.
I20260929 03:24:53.243916 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9722
I20260929 03:24:53.244190 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:34335
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=46227
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:53.249940 9791 log.cc:824] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.252035 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:24:53.252399 9791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap complete.
I20260929 03:24:53.254765 9791 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:53.255005 9791 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfb7488032e4feab656f559b806741e, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.255187 9791 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:53.255311 9791 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:53.255379 9791 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:53.255431 9791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:24:53.256531 9791 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:53.256665 9791 leader_election.cc:302] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfb7488032e4feab656f559b806741e; no voters:
I20260929 03:24:53.256886 9791 leader_election.cc:288] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:24:53.256932 9795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:24:53.257102 9795 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 LEADER]: Becoming Leader. State: Replica: fcfb7488032e4feab656f559b806741e, State: Running, Role: LEADER
I20260929 03:24:53.257184 9791 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:53.257257 9795 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:53.257532 9795 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfb7488032e4feab656f559b806741e. Latest consensus state: current_term: 2 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:53.257612 9795 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:53.257567 9796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:53.257795 9796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:53.258478 9801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:53.259367 9801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:53.259534 9801 catalog_manager.cc:1259] Loaded cluster ID: 68257998dfc446368cda8ee42c9e4d5d
I20260929 03:24:53.259757 9801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:24:53.260061 9801 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:24:53.265133 9801 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:24:53.267798 9801 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:24:53.268019 9801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Loaded TSK: 0
I20260929 03:24:53.268231 9801 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:24:53.331939 9793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:53.332175 9793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:53.332201 9793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:53.333436 9793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:53.333483 9793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:53.333511 9793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:24:53.335079 9793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:34335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=46227
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:53.335384 9793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:53.335645 9793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:53.340737 9821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.340704 9818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.340701 9819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.341107 9793 server_base.cc:1034] running on GCE node
I20260929 03:24:53.341346 9793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:53.341589 9793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:53.342751 9793 hybrid_clock.cc:648] HybridClock initialized: now 1790652293342722 us; error 48 us; skew 500 ppm
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:53.345520 9793 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:24:53.346050 9793 webserver.cc:460] Webserver started at http://127.27.68.65:46227/ using document root <none> and password file <none>
I20260929 03:24:53.346313 9793 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:53.346396 9793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:53.347903 9793 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 03:24:53.348629 9828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.348807 9793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:53.348908 9793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "e68dd44f42ee4e39ab7ac5408a42c394"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "382ad5d12306e0b1a43d5efdcaedad59"
server_key_iv: "2884ca8cf7a3f226a64dec30fe5d2a1c"
server_key_version: "encryptionkey@0"
I20260929 03:24:53.349259 9793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:53.371966 9793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:53.374518 9793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:53.374675 9793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:53.374972 9793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:53.375564 9835 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:24:53.377612 9793 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:24:53.377692 9793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:24:53.377739 9793 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:24:53.378443 9793 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:24:53.378502 9793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:53.378575 9835 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap starting.
I20260929 03:24:53.385950 9793 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:34335
I20260929 03:24:53.386118 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:34335 every 8 connection(s)
I20260929 03:24:53.386412 9793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:24:53.389649 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9793
I20260929 03:24:53.389786 9835 log.cc:824] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.389961 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:34261
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=39857
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:53.392565 9835 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:24:53.392894 9835 tablet_bootstrap.cc:492] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap complete.
I20260929 03:24:53.394057 9835 ts_tablet_manager.cc:1397] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:53.395401 9835 raft_consensus.cc:348] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.395815 9835 raft_consensus.cc:729] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.395995 9835 consensus_queue.cc:260] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } } peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } }
I20260929 03:24:53.396289 9835 ts_tablet_manager.cc:1428] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:24:53.399477 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.387732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:43361 (local address 127.27.68.126:44107)
0929 03:24:53.387909 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.387918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.387956 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:24:53.388594 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.388599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.388645 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.388786 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:53.389452 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.390839 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.392266 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.393117 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.393761 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.395769 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:53.395773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:53.395773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:53.395786 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:53.397616 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.398076 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.398079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.398080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.398181 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.398455 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.398456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.398457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.398695 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:53.398793 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.399101 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.399267 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":111,"threads_started":1}
I20260929 03:24:53.399478 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.387378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:43361)
0929 03:24:53.387546 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.387591 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.388347 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.388816 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.388820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.389133 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:53.389324 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.389330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.390983 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.390985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.392150 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.392154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.393249 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.393251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.393419 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.394086 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:53.394095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:53.395649 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:53.397730 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.397738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.397740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.397984 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.398255 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.398256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.398256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.398378 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.398770 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:53.398778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:53.398930 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.399044 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.399264 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260929 03:24:53.400338 9943 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:53.400458 9943 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:53.400645 9943 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:53.401463 9755 ts_manager.cc:194] Registered new tserver with Master: e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65:34335)
I20260929 03:24:53.402844 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:43361
I20260929 03:24:53.403869 9943 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.65@KRBTEST.COM
I20260929 03:24:53.410843 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.402178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:34335 (local address 127.0.0.1:55008)
0929 03:24:53.402444 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.402483 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.402573 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.402768 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.402773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.403131 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:53.403215 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.403218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.404429 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.404429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.405031 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.405032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.405671 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.405672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.405733 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.406271 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:53.406275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:53.407658 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:53.409635 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.409642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.409644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.409859 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.410080 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.410080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.410081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.410134 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.410502 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:53.410509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:53.410560 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.410645 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.410766 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260929 03:24:53.410941 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.402231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.27.68.65:34335)
0929 03:24:53.402381 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.402386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.402555 (+ 169us) server_negotiation.cc:407] Connection header received
0929 03:24:53.402660 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.402662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.402676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.402752 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:53.403323 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.404341 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.405110 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.405579 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.405663 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.407745 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:53.407755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:53.407756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:53.407764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:53.409542 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.409941 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.409945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.409945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.410013 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.410192 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.410193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.410193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.410390 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:53.410491 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.410710 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.410827 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260929 03:24:53.411425 9877 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b8630004d40748ee48a9ae9ce2431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:55008
I20260929 03:24:53.412081 9952 tablet_replica.cc:330] stopping tablet replica
I20260929 03:24:53.412217 9952 raft_consensus.cc:2229] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:53.412302 9952 raft_consensus.cc:2258] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:53.412631 9952 ts_tablet_manager.cc:1905] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:53.414844 9952 ts_tablet_manager.cc:1918] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:53.414958 9952 log.cc:1196] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/bc0b8630004d40748ee48a9ae9ce2431
I20260929 03:24:53.415194 9952 ts_tablet_manager.cc:1939] T bc0b8630004d40748ee48a9ae9ce2431 P e68dd44f42ee4e39ab7ac5408a42c394: Deleting consensus metadata
I20260929 03:24:53.415778 9740 catalog_manager.cc:4841] TS e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65:34335): tablet bc0b8630004d40748ee48a9ae9ce2431 (table test-table [id=e970a790411542b1bbe38ea509210970]) successfully deleted
W20260929 03:24:53.481299 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:53.481539 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:53.481564 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:53.482827 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:53.482877 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:53.482905 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:24:53.484694 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:34261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=39857
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:53.484997 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:53.485244 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:53.490365 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.490365 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.490513 9948 server_base.cc:1034] running on GCE node
W20260929 03:24:53.490351 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.490778 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:53.491034 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:53.492182 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790652293492160 us; error 43 us; skew 500 ppm
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:53.494704 9948 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:24:53.495296 9948 webserver.cc:460] Webserver started at http://127.27.68.66:39857/ using document root <none> and password file <none>
I20260929 03:24:53.495582 9948 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:53.495669 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:53.497056 9948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:24:53.497690 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.497866 9948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:53.497993 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "5d091f993b0245298709088576cb2ae6"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "f6797c56a420a13636cd380cb32459f5"
server_key_iv: "2b56991f295f679561f6b7a56bc64a37"
server_key_version: "encryptionkey@0"
I20260929 03:24:53.498366 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:53.509299 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:53.511271 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:53.511462 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:53.511724 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:53.512073 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:53.512130 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.512172 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:53.512202 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.518384 9948 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:34261
I20260929 03:24:53.518427 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:34261 every 8 connection(s)
I20260929 03:24:53.518823 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:53.527951 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.519637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51997 (local address 127.27.68.126:44107)
0929 03:24:53.519684 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.519686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.519918 (+ 232us) server_negotiation.cc:407] Connection header received
0929 03:24:53.520451 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.520452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.520465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.520517 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:53.521460 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.521965 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.522984 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.523183 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.523253 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.525465 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:53.525466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:53.525466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:53.525473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:53.526671 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.527048 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.527049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.527049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.527096 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.527374 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.527375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.527375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.527504 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:53.527564 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.527806 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.527881 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:24:53.528179 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.519645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:51997)
0929 03:24:53.519807 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.519844 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.520327 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.520573 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.520576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.521148 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:53.521359 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.521364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.522063 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.522064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.522900 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.522902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.523334 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.523334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.523452 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.523958 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:53.523967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:53.525361 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:53.526757 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.526763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.526765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.526983 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.527153 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.527153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.527153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.527317 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.527566 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:53.527572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:53.527694 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.527782 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.527915 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":113,"threads_started":1}
I20260929 03:24:53.528716 10081 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:53.528827 10081 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:53.529006 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 9948
I20260929 03:24:53.529035 10081 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:53.529328 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:45743
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=44157
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:53.529563 9755 ts_manager.cc:194] Registered new tserver with Master: 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
I20260929 03:24:53.529947 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:51997
W20260929 03:24:53.613022 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:53.613243 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:53.613268 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:53.614532 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:53.614590 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:53.614617 10084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:24:53.616147 10084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:45743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=44157
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:53.616353 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:53.616570 10084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:53.621702 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.621840 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.621708 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.622153 10084 server_base.cc:1034] running on GCE node
I20260929 03:24:53.622380 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:53.622598 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:53.623737 10084 hybrid_clock.cc:648] HybridClock initialized: now 1790652293623713 us; error 41 us; skew 500 ppm
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:53.626220 10084 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:24:53.626705 10084 webserver.cc:460] Webserver started at http://127.27.68.67:44157/ using document root <none> and password file <none>
I20260929 03:24:53.626938 10084 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:53.627013 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:53.628474 10084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:53.629093 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.629278 10084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:53.629371 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "a5525e1168764494afa4143a6d73632e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "98b1c613cfd4c9e1d120bc1e1e32d8c4"
server_key_iv: "6667bd40942d271f2ab30aa940cefea1"
server_key_version: "encryptionkey@0"
I20260929 03:24:53.629724 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:53.661103 10084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:53.663092 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:53.663288 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:53.663565 10084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:53.663892 10084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:53.663933 10084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.663986 10084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:53.664016 10084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.669970 10084 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:45743
I20260929 03:24:53.670033 10212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:45743 every 8 connection(s)
I20260929 03:24:53.670378 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:24:53.675002 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10084
Sep 29 03:24:53 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:24:53.680714 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.671196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:54761 (local address 127.27.68.126:44107)
0929 03:24:53.671264 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.671267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.671493 (+ 226us) server_negotiation.cc:407] Connection header received
0929 03:24:53.672057 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.672058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.672072 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.672141 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:53.673138 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.673645 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.674680 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.674918 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.675009 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.677454 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:53.677455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:53.677456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:53.677463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:53.679033 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.679597 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.679598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.679599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.679664 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.679963 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.679963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.679964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.680112 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:53.680181 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.680494 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.680588 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:24:53.680872 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.671164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:54761)
0929 03:24:53.671358 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.671396 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.671920 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.672221 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.672224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.672802 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:53.673010 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.673016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.673749 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.673750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.674570 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.674572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.675019 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.675021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.675184 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.675836 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:53.675846 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:53.677332 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:53.679119 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.679127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.679129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.679505 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.679739 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:53.679739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:53.679739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:53.679847 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:53.680185 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:53.680191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:53.680320 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.680447 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.680638 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":147,"threads_started":1}
I20260929 03:24:53.681398 10213 heartbeater.cc:347] Connected to a master server at 127.27.68.126:44107
I20260929 03:24:53.681510 10213 heartbeater.cc:464] Registering TS with master...
I20260929 03:24:53.681718 10213 heartbeater.cc:511] Master 127.27.68.126:44107 requested a full tablet report, sending...
I20260929 03:24:53.682174 9755 ts_manager.cc:194] Registered new tserver with Master: a5525e1168764494afa4143a6d73632e (127.27.68.67:45743)
I20260929 03:24:53.682565 9755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:54761
I20260929 03:24:53.682765 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.675602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.27.68.126:44107)
0929 03:24:53.675736 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.675738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.675745 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:24:53.675773 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.675774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.675783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.675866 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:53.676468 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.676939 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.677904 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.678223 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.678301 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.679625 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:53.679627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:53.679627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:53.679633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:53.681426 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.681847 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.681848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.681849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.681909 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:53.682122 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:53.682123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:53.682123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:53.682230 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:53.682417 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.682455 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.682535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260929 03:24:53.683487 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:24:53.686412 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.683850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.27.68.126:44107)
0929 03:24:53.683935 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.683937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.684017 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:24:53.684101 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.684102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.684138 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.684220 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:24:53.684348 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.684830 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.685604 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.685820 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.685903 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.686079 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:24:53.686187 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:24:53.686274 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.686313 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.686336 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:24:53.686739 9755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:24:53.688421 9755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:24:53.690618 9877 tablet_service.cc:1467] Processing CreateTablet for tablet e626455021c043ed920f115d9f3aa8dc (DEFAULT_TABLE table=test-table [id=730a99f481cd4401af8602e621d98d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.690851 9877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e626455021c043ed920f115d9f3aa8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.692515 9835 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap starting.
I20260929 03:24:53.692837 9835 tablet_bootstrap.cc:654] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.694481 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.690409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:34261 (local address 127.0.0.1:37590)
0929 03:24:53.690676 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.690686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.690736 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.691148 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.691149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.691151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.691186 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.691187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.692166 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.692167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.693478 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.693481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.694175 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.694177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.694260 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.694264 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.694320 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.694363 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20260929 03:24:53.694523 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.690402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.27.68.67:45743)
0929 03:24:53.690853 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.690858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.690869 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:24:53.690909 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.690911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.690929 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.691042 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.691259 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.692392 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.693532 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.694017 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.694106 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.694167 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.694414 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.694458 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":130,"threads_started":1}
I20260929 03:24:53.694576 9835 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: No bootstrap required, opened a new log
I20260929 03:24:53.694618 9835 ts_tablet_manager.cc:1397] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:24:53.694722 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.690340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:45743 (local address 127.0.0.1:43452)
0929 03:24:53.690386 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.690397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.690433 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.691054 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.691055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.691057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.691120 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.691121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.692518 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.692519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.693456 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.693458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.694104 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.694105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.694174 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.694198 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.694257 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.694312 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":7}
I20260929 03:24:53.694761 9835 raft_consensus.cc:348] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.694844 9835 raft_consensus.cc:374] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.694890 9835 raft_consensus.cc:729] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.694949 9835 consensus_queue.cc:260] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.695052 9835 ts_tablet_manager.cc:1428] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:53.695171 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.690663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.27.68.66:34261)
0929 03:24:53.690892 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.690896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.690932 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:24:53.690973 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.690975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.690991 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.691069 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.691302 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.692059 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.693569 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.694070 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.694169 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.694229 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.694336 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.694380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260929 03:24:53.695509 10147 tablet_service.cc:1467] Processing CreateTablet for tablet e626455021c043ed920f115d9f3aa8dc (DEFAULT_TABLE table=test-table [id=730a99f481cd4401af8602e621d98d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.695868 10147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e626455021c043ed920f115d9f3aa8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.695832 10015 tablet_service.cc:1467] Processing CreateTablet for tablet e626455021c043ed920f115d9f3aa8dc (DEFAULT_TABLE table=test-table [id=730a99f481cd4401af8602e621d98d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:24:53.696139 10015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e626455021c043ed920f115d9f3aa8dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:24:53.699571 10223 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Bootstrap starting.
I20260929 03:24:53.699779 10224 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Bootstrap starting.
I20260929 03:24:53.700381 10224 tablet_bootstrap.cc:654] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.700910 10224 log.cc:824] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.701692 10224 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: No bootstrap required, opened a new log
I20260929 03:24:53.701769 10223 tablet_bootstrap.cc:654] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Neither blocks nor log segments found. Creating new log.
I20260929 03:24:53.701783 10224 ts_tablet_manager.cc:1397] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:24:53.702368 10223 log.cc:824] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:53.703050 10223 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: No bootstrap required, opened a new log
I20260929 03:24:53.703102 10223 ts_tablet_manager.cc:1397] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 03:24:53.703454 10224 raft_consensus.cc:348] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.703606 10224 raft_consensus.cc:374] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.703650 10224 raft_consensus.cc:729] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5525e1168764494afa4143a6d73632e, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.703811 10224 consensus_queue.cc:260] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.704093 10224 ts_tablet_manager.cc:1428] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:24:53.704113 10213 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
I20260929 03:24:53.705340 10223 raft_consensus.cc:348] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.705466 10223 raft_consensus.cc:374] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:24:53.705487 10223 raft_consensus.cc:729] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d091f993b0245298709088576cb2ae6, State: Initialized, Role: FOLLOWER
I20260929 03:24:53.705618 10223 consensus_queue.cc:260] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.705919 10081 heartbeater.cc:503] Master 127.27.68.126:44107 was elected leader, sending a full tablet report...
I20260929 03:24:53.705919 10223 ts_tablet_manager.cc:1428] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 03:24:53.769495 10082 tablet.cc:2369] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:24:53.847589 9949 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:53.847741 9949 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.848099 9949 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
I20260929 03:24:53.852138 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.848373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:55729 (local address 127.27.68.66:34261)
0929 03:24:53.848431 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.848433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.848612 (+ 179us) server_negotiation.cc:407] Connection header received
0929 03:24:53.848644 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.848645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.848654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.848710 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.848880 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.849619 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851090 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.851711 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851784 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.851795 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.851990 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.852034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:24:53.852138 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.848411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:33909 (local address 127.27.68.67:45743)
0929 03:24:53.848462 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.848464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.848476 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:24:53.848523 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.848526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.848541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.848620 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:24:53.848852 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.849619 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851089 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.851708 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851785 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.851795 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.851995 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.852034 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:24:53.852141 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.848291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:45743 (local address 127.27.68.65:33909)
0929 03:24:53.848368 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.848381 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.848446 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.848657 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.848658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.848665 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.848755 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.848757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.849716 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.849717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.850989 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.850993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.851792 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.851874 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.851901 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.851984 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.852016 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":30}
I20260929 03:24:53.852154 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.848296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:34261 (local address 127.27.68.65:55729)
0929 03:24:53.848511 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.848521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:53.848574 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:53.848726 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:53.848726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:53.848728 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:24:53.848796 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.848798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.849717 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.849717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.850989 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:53.850993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.851807 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.851807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:53.851871 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.851902 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 03:24:53.851966 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:24:53.852014 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":128,"thread_start_us":100,"threads_started":1}
I20260929 03:24:53.852562 10035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e626455021c043ed920f115d9f3aa8dc" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d091f993b0245298709088576cb2ae6" is_pre_election: true
I20260929 03:24:53.852586 10167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e626455021c043ed920f115d9f3aa8dc" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5525e1168764494afa4143a6d73632e" is_pre_election: true
I20260929 03:24:53.852747 10167 raft_consensus.cc:2454] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 0.
I20260929 03:24:53.852747 10035 raft_consensus.cc:2454] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 0.
I20260929 03:24:53.852980 9831 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5525e1168764494afa4143a6d73632e, e68dd44f42ee4e39ab7ac5408a42c394; no voters:
I20260929 03:24:53.853160 9949 raft_consensus.cc:2781] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:24:53.853211 9949 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:24:53.853228 9949 raft_consensus.cc:3037] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.853941 9949 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.854094 9949 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 election: Requested vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
I20260929 03:24:53.854274 10167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e626455021c043ed920f115d9f3aa8dc" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5525e1168764494afa4143a6d73632e"
I20260929 03:24:53.854357 10167 raft_consensus.cc:3037] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.855067 10167 raft_consensus.cc:2454] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 1.
I20260929 03:24:53.855195 10035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e626455021c043ed920f115d9f3aa8dc" candidate_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d091f993b0245298709088576cb2ae6"
I20260929 03:24:53.855258 9831 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5525e1168764494afa4143a6d73632e, e68dd44f42ee4e39ab7ac5408a42c394; no voters:
I20260929 03:24:53.855304 10035 raft_consensus.cc:3037] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:24:53.855356 9949 raft_consensus.cc:2781] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:24:53.855471 9949 raft_consensus.cc:686] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 LEADER]: Becoming Leader. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Running, Role: LEADER
I20260929 03:24:53.855551 9949 consensus_queue.cc:237] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:53.856180 10035 raft_consensus.cc:2454] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e68dd44f42ee4e39ab7ac5408a42c394 in term 1.
I20260929 03:24:53.856501 9755 catalog_manager.cc:5495] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 reported cstate change: term changed from 0 to 1, leader changed from <none> to e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } health_report { overall_health: HEALTHY } } }
I20260929 03:24:53.875934 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.873668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.27.68.65:34335)
0929 03:24:53.873733 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:53.873736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:53.873743 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:24:53.873782 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:53.873783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:53.873802 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:53.873866 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:24:53.873973 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.874475 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.875170 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:53.875438 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:53.875493 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:53.875609 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:24:53.875708 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:24:53.875759 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:53.875857 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:24:53.875874 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:24:53.877501 10035 raft_consensus.cc:1261] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Refusing update from remote peer e68dd44f42ee4e39ab7ac5408a42c394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:53.877511 10167 raft_consensus.cc:1261] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Refusing update from remote peer e68dd44f42ee4e39ab7ac5408a42c394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:24:53.877966 10230 consensus_queue.cc:1035] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:53.878127 9949 consensus_queue.cc:1035] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:24:53.881349 10234 mvcc.cc:205] Tried to move back new op lower bound from 7334511795719786496 to 7334511795633164288. Current Snapshot: MvccSnapshot[applied={T|T < 7334511795719786496 or (T in {7334511795719786496})}]
I20260929 03:24:53.881811 10232 mvcc.cc:205] Tried to move back new op lower bound from 7334511795719786496 to 7334511795633164288. Current Snapshot: MvccSnapshot[applied={T|T < 7334511795719786496 or (T in {7334511795719786496})}]
I20260929 03:24:53.883116 9754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:53.883253 9754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:24:53.884059 9754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Sending DeleteTablet for 3 replicas of tablet e626455021c043ed920f115d9f3aa8dc
I20260929 03:24:53.884402 10015 tablet_service.cc:1514] Processing DeleteTablet for tablet e626455021c043ed920f115d9f3aa8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:37590
I20260929 03:24:53.884483 10147 tablet_service.cc:1514] Processing DeleteTablet for tablet e626455021c043ed920f115d9f3aa8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:43452
I20260929 03:24:53.884583 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9793
I20260929 03:24:53.884604 9877 tablet_service.cc:1514] Processing DeleteTablet for tablet e626455021c043ed920f115d9f3aa8dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:24:53 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:55008
I20260929 03:24:53.885522 10242 tablet_replica.cc:330] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: stopping tablet replica
I20260929 03:24:53.885633 10242 raft_consensus.cc:2229] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:53.885742 10242 raft_consensus.cc:2258] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:53.886212 10242 ts_tablet_manager.cc:1905] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:53.888237 10242 ts_tablet_manager.cc:1918] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:53.888305 10242 log.cc:1196] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/e626455021c043ed920f115d9f3aa8dc
I20260929 03:24:53.888602 10242 ts_tablet_manager.cc:1939] T e626455021c043ed920f115d9f3aa8dc P 5d091f993b0245298709088576cb2ae6: Deleting consensus metadata
I20260929 03:24:53.889086 9739 catalog_manager.cc:4841] TS 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): tablet e626455021c043ed920f115d9f3aa8dc (table test-table [id=730a99f481cd4401af8602e621d98d35]) successfully deleted
W20260929 03:24:53.890815 9740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:34335 (error 108)
I20260929 03:24:53.891383 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:53.891267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:34335 (local address 127.0.0.1:55034)
0929 03:24:53.891307 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:53.891345 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:24:53.891412 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9948
I20260929 03:24:53.891438 10243 tablet_replica.cc:330] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: stopping tablet replica
W20260929 03:24:53.891510 9740 catalog_manager.cc:4584] TS e68dd44f42ee4e39ab7ac5408a42c394 (127.27.68.65:34335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e626455021c043ed920f115d9f3aa8dc: Network error: Client connection negotiation failed: client connection to 127.27.68.65:34335: connect: Connection refused (error 111)
I20260929 03:24:53.892206 10243 raft_consensus.cc:2229] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:24:53.892309 10243 raft_consensus.cc:2258] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:24:53.892719 10243 ts_tablet_manager.cc:1905] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:24:53.894023 10243 ts_tablet_manager.cc:1918] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:24:53.894090 10243 log.cc:1196] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/e626455021c043ed920f115d9f3aa8dc
I20260929 03:24:53.894397 10243 ts_tablet_manager.cc:1939] T e626455021c043ed920f115d9f3aa8dc P a5525e1168764494afa4143a6d73632e: Deleting consensus metadata
I20260929 03:24:53.894841 9741 catalog_manager.cc:4841] TS a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): tablet e626455021c043ed920f115d9f3aa8dc (table test-table [id=730a99f481cd4401af8602e621d98d35]) successfully deleted
I20260929 03:24:53.896879 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10084
I20260929 03:24:53.901392 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 9722
I20260929 03:24:53.906797 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--webserver_interface=127.27.68.126
--webserver_port=33623
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:53.990746 10245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:53.990967 10245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:53.991003 10245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:53.992306 10245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:24:53.992352 10245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:53.992367 10245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:53.992379 10245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:24:53.992448 10245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:24:53.993963 10245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=33623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:53.994230 10245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:53.994518 10245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:53.999560 10250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.999562 10253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:53.999570 10251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:53.999730 10245 server_base.cc:1034] running on GCE node
I20260929 03:24:54.000151 10245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:54.000505 10245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:54.001672 10245 hybrid_clock.cc:648] HybridClock initialized: now 1790652294001646 us; error 37 us; skew 500 ppm
Sep 29 03:24:54 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:54.004273 10245 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:24:54.004738 10245 webserver.cc:460] Webserver started at http://127.27.68.126:33623/ using document root <none> and password file <none>
I20260929 03:24:54.004944 10245 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:54.004987 10245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:54.006296 10245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:54.006979 10260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.007277 10245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:54.007356 10245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fcfb7488032e4feab656f559b806741e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "2f7edd95289f167ed31a717906efe6f2"
server_key_iv: "b0735d424d0a12cc1c4089d3c6d58611"
server_key_version: "encryptionkey@0"
I20260929 03:24:54.007711 10245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:54.021139 10245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:54.023015 10245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:54.023133 10245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:54.027156 10245 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:44107
I20260929 03:24:54.027189 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:44107 every 8 connection(s)
I20260929 03:24:54.027663 10245 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:24:54.029239 10313 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:24:54.029932 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap starting.
I20260929 03:24:54.031862 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10245
I20260929 03:24:54.032176 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:34335
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=46227
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:54.038540 10313 log.cc:824] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:54.041003 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:24:54.041344 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e: Bootstrap complete.
I20260929 03:24:54.043655 10313 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:54.043913 10313 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfb7488032e4feab656f559b806741e, State: Initialized, Role: FOLLOWER
I20260929 03:24:54.044108 10313 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:54.044212 10313 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:24:54.044294 10313 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:24:54.044346 10313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:24:54.045413 10313 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:54.045543 10313 leader_election.cc:302] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfb7488032e4feab656f559b806741e; no voters:
I20260929 03:24:54.045764 10313 leader_election.cc:288] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:24:54.045802 10317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:24:54.045907 10317 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [term 3 LEADER]: Becoming Leader. State: Replica: fcfb7488032e4feab656f559b806741e, State: Running, Role: LEADER
I20260929 03:24:54.046007 10317 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } }
I20260929 03:24:54.046066 10313 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:24:54.046257 10317 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfb7488032e4feab656f559b806741e. Latest consensus state: current_term: 3 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:54.046317 10317 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:54.046463 10318 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fcfb7488032e4feab656f559b806741e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfb7488032e4feab656f559b806741e" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 44107 } } }
I20260929 03:24:54.046629 10318 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfb7488032e4feab656f559b806741e [sys.catalog]: This master's current role is: LEADER
I20260929 03:24:54.046514 10321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:24:54.047606 10321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:24:54.047772 10321 catalog_manager.cc:1259] Loaded cluster ID: 68257998dfc446368cda8ee42c9e4d5d
I20260929 03:24:54.048012 10321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 03:24:54.048375 10321 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6e445579ad google::LogMessage::Fail() at ??:0
@ 0x7f6e4455a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6e445574de google::LogMessage::Flush() at ??:0
@ 0x7f6e44558a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6e4c47c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6e4c4858b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6e464250d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6e4641a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6e4727e6db start_thread at ??:0
@ 0x7f6e4305561f clone at ??:0
W20260929 03:24:54.129714 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:54.129935 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:54.129961 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:54.131192 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:54.131304 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:54.131358 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:24:54.132915 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:34335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=46227
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:54.133170 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:54.133441 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:54.138597 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:54.138617 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:54.138617 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:54.138904 10315 server_base.cc:1034] running on GCE node
I20260929 03:24:54.139111 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:54.139416 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:54.140578 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790652294140560 us; error 30 us; skew 500 ppm
Sep 29 03:24:54 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:54.143040 10315 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:24:54.143563 10315 webserver.cc:460] Webserver started at http://127.27.68.65:46227/ using document root <none> and password file <none>
I20260929 03:24:54.143826 10315 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:54.143872 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:54.145236 10315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:24:54.145943 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.146099 10315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:24:54.146185 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "e68dd44f42ee4e39ab7ac5408a42c394"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "382ad5d12306e0b1a43d5efdcaedad59"
server_key_iv: "2884ca8cf7a3f226a64dec30fe5d2a1c"
server_key_version: "encryptionkey@0"
I20260929 03:24:54.146529 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:54.188532 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:54.190874 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:54.191023 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:54.191339 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:54.191821 10366 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:24:54.193914 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:24:54.194013 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:24:54.194175 10315 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:24:54.194893 10315 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:24:54.194962 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:54.195036 10366 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap starting.
I20260929 03:24:54.202898 10315 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:34335
I20260929 03:24:54.202988 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:34335 every 8 connection(s)
I20260929 03:24:54.203435 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:24:54.206501 10366 log.cc:824] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Log is configured to *not* fsync() on all Append() calls
I20260929 03:24:54.207968 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10315
I20260929 03:24:54.208244 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:34261
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=39857
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:24:54.209638 10366 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:24:54.209960 10366 tablet_bootstrap.cc:492] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Bootstrap complete.
I20260929 03:24:54.211182 10366 ts_tablet_manager.cc:1397] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260929 03:24:54.212679 10366 raft_consensus.cc:348] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:54.213017 10366 raft_consensus.cc:729] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e68dd44f42ee4e39ab7ac5408a42c394, State: Initialized, Role: FOLLOWER
I20260929 03:24:54.213152 10366 consensus_queue.cc:260] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:54.213423 10366 ts_tablet_manager.cc:1428] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260929 03:24:54.274807 10476 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:24:54.204202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:56599)
0929 03:24:54.204405 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.204444 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:54.204924 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:54.274594 (+ 69670us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
W20260929 03:24:54.275115 10474 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:44107 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:44107: Client connection negotiation failed: client connection to 127.27.68.126:44107: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:24:54.275543 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.275421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:36661)
0929 03:24:54.275477 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.275522 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:24:54.275971 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.275886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:52701)
0929 03:24:54.275933 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.275957 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:24:54.276105 10474 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:24:54.297793 10478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:54.298065 10478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:54.298111 10478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:54.299409 10478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:54.299482 10478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:54.299533 10478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:24:54.301126 10478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:34261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=39857
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:54.301445 10478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:54.301724 10478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:54.306945 10485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:54.306927 10484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:54.307039 10478 server_base.cc:1034] running on GCE node
W20260929 03:24:54.306927 10487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:54.307502 10478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:54.307772 10478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:54.308940 10478 hybrid_clock.cc:648] HybridClock initialized: now 1790652294308930 us; error 31 us; skew 500 ppm
Sep 29 03:24:54 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:54.311407 10478 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:24:54.311857 10478 webserver.cc:460] Webserver started at http://127.27.68.66:39857/ using document root <none> and password file <none>
I20260929 03:24:54.312147 10478 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:54.312223 10478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:54.313627 10478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:54.314244 10494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.314529 10478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:24:54.314634 10478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "5d091f993b0245298709088576cb2ae6"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "f6797c56a420a13636cd380cb32459f5"
server_key_iv: "2b56991f295f679561f6b7a56bc64a37"
server_key_version: "encryptionkey@0"
I20260929 03:24:54.315032 10478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:54.337738 10478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:54.339685 10478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:54.339856 10478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:54.340101 10478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:54.340421 10478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:54.340476 10478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.340515 10478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:54.340541 10478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.346509 10478 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:34261
I20260929 03:24:54.346556 10607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:34261 every 8 connection(s)
I20260929 03:24:54.346909 10478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:24:54.347949 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.347579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:52391)
0929 03:24:54.347726 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.347811 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260929 03:24:54.348343 10608 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:44107 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:44107: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
I20260929 03:24:54.348632 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.348531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:50205)
0929 03:24:54.348583 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.348613 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:24:54.348927 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.348840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:33051)
0929 03:24:54.348883 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.348907 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:24:54.349025 10608 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:24:54.355741 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10478
I20260929 03:24:54.356019 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:45743
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:44107
--webserver_port=44157
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:24:54.439078 10611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:24:54.439320 10611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:24:54.439347 10611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:24:54.440567 10611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:24:54.440611 10611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:24:54.440652 10611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:24:54.442114 10611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:45743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=44157
--tserver_master_addrs=127.27.68.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:24:54.442306 10611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:24:54.442485 10611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:24:54.447605 10616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:54.447628 10619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:24:54.447605 10617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:24:54.447830 10611 server_base.cc:1034] running on GCE node
I20260929 03:24:54.448057 10611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:24:54.448298 10611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:24:54.449456 10611 hybrid_clock.cc:648] HybridClock initialized: now 1790652294449445 us; error 29 us; skew 500 ppm
Sep 29 03:24:54 dist-test-slave-fzwk krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:24:54.452356 10611 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:24:54.452804 10611 webserver.cc:460] Webserver started at http://127.27.68.67:44157/ using document root <none> and password file <none>
I20260929 03:24:54.453039 10611 fs_manager.cc:359] Metadata directory not provided
I20260929 03:24:54.453087 10611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:24:54.454479 10611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:24:54.455093 10626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.455279 10611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:24:54.455382 10611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "a5525e1168764494afa4143a6d73632e"
format_stamp: "Formatted at 2026-09-29 03:24:52 on dist-test-slave-fzwk"
server_key: "98b1c613cfd4c9e1d120bc1e1e32d8c4"
server_key_iv: "6667bd40942d271f2ab30aa940cefea1"
server_key_version: "encryptionkey@0"
I20260929 03:24:54.455736 10611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:24:54.468761 10611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:24:54.471169 10611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:24:54.471360 10611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:24:54.471570 10611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:24:54.471832 10611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:24:54.471863 10611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.471885 10611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:24:54.471899 10611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:24:54.477730 10611 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:45743
I20260929 03:24:54.477818 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:45743 every 8 connection(s)
I20260929 03:24:54.478183 10611 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:24:54.479422 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.478896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:40153)
0929 03:24:54.479080 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.479199 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
W20260929 03:24:54.479911 10740 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:44107 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:44107: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
I20260929 03:24:54.480222 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.480128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:49813)
0929 03:24:54.480173 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.480203 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:24:54.480482 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:54.480432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:58379)
0929 03:24:54.480452 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:54.480470 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260929 03:24:54.480592 10740 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:24:54.481181 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10611
I20260929 03:24:55.276682 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:55.276466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:59461)
0929 03:24:55.276586 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:55.276621 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 03:24:55.349596 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:55.349352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:55005)
0929 03:24:55.349495 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:55.349547 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":53,"threads_started":1}
I20260929 03:24:55.481221 10746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:55.481013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:41841)
0929 03:24:55.481134 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:55.481170 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260929 03:24:56.078370 10747 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:56.078577 10747 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:56.078961 10747 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
Sep 29 03:24:56 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.67@KRBTEST.COM
Sep 29 03:24:56 dist-test-slave-fzwk krb5kdc[9175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.66@KRBTEST.COM
I20260929 03:24:56.089774 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.079376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:53701 (local address 127.27.68.66:34261)
0929 03:24:56.079503 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:56.079517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:56.079545 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:24:56.080110 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:56.080112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:56.080157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:56.080313 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:56.081058 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.081925 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.082934 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.083589 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.083701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:56.086403 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:56.086406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:56.086407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:56.086418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:56.088048 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088508 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:56.088513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:56.088513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:56.088610 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088875 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:56.088876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:56.088877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:56.089135 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:56.089231 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:56.089507 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:24:56.089651 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260929 03:24:56.089762 10748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.079176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:45743 (local address 127.27.68.65:52145)
0929 03:24:56.079503 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:56.079514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:56.079574 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:56.080375 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:56.080376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:56.080752 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:56.080939 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:56.080944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.082054 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.082055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:56.082867 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:56.082870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.083815 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.083816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:56.083860 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:56.084424 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:56.084429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:56.085900 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:56.088061 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:56.088071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:56.088405 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:56.088688 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:56.088689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:56.088808 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:56.089220 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:56.089221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:56.089342 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 03:24:56.089452 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:24:56.089663 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":56,"thread_start_us":102,"threads_started":1}
I20260929 03:24:56.089793 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.079317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52145 (local address 127.27.68.67:45743)
0929 03:24:56.079576 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 03:24:56.079587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:24:56.079611 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:24:56.080119 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:24:56.080122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:24:56.080161 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:24:56.080273 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:24:56.081067 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.081924 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.082968 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.083590 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.083701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:56.086044 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:24:56.086049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:24:56.086049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:24:56.086060 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:24:56.087969 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088498 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:56.088502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:56.088503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:56.088612 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088886 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:24:56.088887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:24:56.088887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:24:56.089135 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:24:56.089228 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:24:56.089513 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 03:24:56.089655 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":51,"threads_started":1}
I20260929 03:24:56.090394 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.079173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:34261 (local address 127.27.68.65:53701)
0929 03:24:56.079326 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:56.079339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:24:56.079405 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:24:56.080368 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:24:56.080370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:24:56.080752 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:24:56.080939 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:56.080944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.082035 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.082036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:56.082817 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:24:56.082840 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:24:56.083674 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:24:56.083675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:24:56.083775 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:24:56.084269 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:24:56.084277 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:24:56.086298 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:24:56.088132 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:56.088133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:56.088406 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:56.088686 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:24:56.088687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:24:56.088688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:24:56.088807 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:24:56.089214 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:24:56.089220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:24:56.089341 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 03:24:56.089452 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:24:56.089749 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":150,"thread_start_us":86,"threads_started":1}
W20260929 03:24:56.090854 10362 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
W20260929 03:24:56.091089 10360 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
I20260929 03:24:56.091151 10360 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68dd44f42ee4e39ab7ac5408a42c394; no voters: 5d091f993b0245298709088576cb2ae6, a5525e1168764494afa4143a6d73632e
I20260929 03:24:56.091292 10747 raft_consensus.cc:2726] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:24:56.277475 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.277276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:35275)
0929 03:24:56.277385 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:56.277434 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 03:24:56.350260 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.350045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:41515)
0929 03:24:56.350161 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:56.350213 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260929 03:24:56.482079 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:56.481833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:59735)
0929 03:24:56.481975 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:56.482030 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260929 03:24:57.278165 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:57.277955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:50679)
0929 03:24:57.278086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:57.278118 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260929 03:24:57.350973 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:57.350779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:37169)
0929 03:24:57.350895 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:57.350928 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260929 03:24:57.482931 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:57.482682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:37097)
0929 03:24:57.482840 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:57.482885 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260929 03:24:58.253587 10757 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:24:58.253710 10757 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:24:58.253913 10757 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
W20260929 03:24:58.254343 10360 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
W20260929 03:24:58.254431 10362 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
I20260929 03:24:58.254458 10360 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68dd44f42ee4e39ab7ac5408a42c394; no voters: 5d091f993b0245298709088576cb2ae6, a5525e1168764494afa4143a6d73632e
I20260929 03:24:58.254536 10757 raft_consensus.cc:2726] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:24:58.278993 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:58.278731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:39615)
0929 03:24:58.278887 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:58.278933 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260929 03:24:58.351666 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:58.351442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:45903)
0929 03:24:58.351560 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:58.351606 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260929 03:24:58.483678 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:58.483463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:51119)
0929 03:24:58.483578 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:58.483618 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260929 03:24:59.279748 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:59.279524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:46391)
0929 03:24:59.279661 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:59.279699 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260929 03:24:59.352324 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:59.352117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:46557)
0929 03:24:59.352245 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:59.352281 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260929 03:24:59.484427 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:24:59.484209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:33483)
0929 03:24:59.484332 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:24:59.484379 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260929 03:25:00.280637 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:00.280326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:53311)
0929 03:25:00.280527 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:00.280566 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260929 03:25:00.353019 10765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:00.352801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:54579)
0929 03:25:00.352935 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:00.352976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260929 03:25:00.485095 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:00.484885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:56859)
0929 03:25:00.485006 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:00.485050 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260929 03:25:01.281549 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:01.281313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:40385)
0929 03:25:01.281455 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:01.281499 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260929 03:25:01.353722 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:01.353514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:39139)
0929 03:25:01.353637 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:01.353674 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260929 03:25:01.475135 10769 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:01.475282 10769 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:25:01.475450 10769 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
W20260929 03:25:01.475843 10362 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
W20260929 03:25:01.475932 10360 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
I20260929 03:25:01.475996 10362 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68dd44f42ee4e39ab7ac5408a42c394; no voters: 5d091f993b0245298709088576cb2ae6, a5525e1168764494afa4143a6d73632e
I20260929 03:25:01.476087 10769 raft_consensus.cc:2726] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:01.485850 10770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:01.485620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:51267)
0929 03:25:01.485757 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:01.485803 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260929 03:25:02.282285 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:02.282051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:60263)
0929 03:25:02.282190 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:02.282236 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260929 03:25:02.354517 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:02.354259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:50409)
0929 03:25:02.354423 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:02.354471 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260929 03:25:02.486603 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:02.486399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:36381)
0929 03:25:02.486518 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:02.486557 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260929 03:25:03.283303 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:03.282991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:59359)
0929 03:25:03.283154 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:03.283202 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260929 03:25:03.355189 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:03.354946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:33055)
0929 03:25:03.355099 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:03.355141 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260929 03:25:03.487511 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:03.487270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:38727)
0929 03:25:03.487409 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:03.487460 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260929 03:25:03.556614 10777 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:03.556730 10777 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:25:03.556932 10777 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
W20260929 03:25:03.557294 10362 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
W20260929 03:25:03.557379 10360 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
I20260929 03:25:03.557472 10360 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68dd44f42ee4e39ab7ac5408a42c394; no voters: 5d091f993b0245298709088576cb2ae6, a5525e1168764494afa4143a6d73632e
I20260929 03:25:03.557556 10777 raft_consensus.cc:2726] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:04.283980 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:04.283752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:57901)
0929 03:25:04.283872 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:04.283912 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260929 03:25:04.355827 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:04.355620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:48991)
0929 03:25:04.355743 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:04.355780 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260929 03:25:04.488361 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:04.488117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:33603)
0929 03:25:04.488272 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:04.488313 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260929 03:25:05.284760 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:05.284503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:56065)
0929 03:25:05.284662 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:05.284710 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260929 03:25:05.356544 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:05.356314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:51163)
0929 03:25:05.356450 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:05.356494 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260929 03:25:05.489037 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:05.488810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:54603)
0929 03:25:05.488947 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:05.488987 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260929 03:25:06.285589 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:06.285328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:47653)
0929 03:25:06.285485 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:06.285538 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260929 03:25:06.357275 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:06.357056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:36481)
0929 03:25:06.357183 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:06.357226 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260929 03:25:06.489738 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:06.489527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:58303)
0929 03:25:06.489660 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:06.489690 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260929 03:25:07.286306 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:07.286074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:43919)
0929 03:25:07.286214 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:07.286256 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260929 03:25:07.357942 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:07.357704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:60201)
0929 03:25:07.357864 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:07.357897 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":54,"threads_started":1}
I20260929 03:25:07.490540 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:07.490294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:46139)
0929 03:25:07.490437 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:07.490491 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260929 03:25:08.016538 10790 raft_consensus.cc:482] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:08.016649 10790 raft_consensus.cc:504] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5525e1168764494afa4143a6d73632e" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 45743 } } peers { permanent_uuid: "5d091f993b0245298709088576cb2ae6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 34261 } } peers { permanent_uuid: "e68dd44f42ee4e39ab7ac5408a42c394" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 34335 } }
I20260929 03:25:08.016857 10790 leader_election.cc:288] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5525e1168764494afa4143a6d73632e (127.27.68.67:45743), 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261)
W20260929 03:25:08.017254 10360 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5d091f993b0245298709088576cb2ae6 (127.27.68.66:34261): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
W20260929 03:25:08.017366 10362 leader_election.cc:341] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5525e1168764494afa4143a6d73632e (127.27.68.67:45743): Not found: Tablet not found: e626455021c043ed920f115d9f3aa8dc
I20260929 03:25:08.017449 10362 leader_election.cc:302] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e68dd44f42ee4e39ab7ac5408a42c394; no voters: 5d091f993b0245298709088576cb2ae6, a5525e1168764494afa4143a6d73632e
I20260929 03:25:08.017557 10790 raft_consensus.cc:2726] T e626455021c043ed920f115d9f3aa8dc P e68dd44f42ee4e39ab7ac5408a42c394 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:08.287097 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:08.286871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:52437)
0929 03:25:08.287002 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:08.287048 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260929 03:25:08.358676 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:08.358452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:49591)
0929 03:25:08.358581 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:08.358626 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260929 03:25:08.491256 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:08.491014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.67:44047)
0929 03:25:08.491137 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:08.491171 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260929 03:25:09.287953 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:09.287731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.65:56127)
0929 03:25:09.287865 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:09.287908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260929 03:25:09.359407 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:09.359171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:44107 (local address 127.27.68.66:58741)
0929 03:25:09.359322 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:09.359360 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:44107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260929 03:25:09.481900 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10315
I20260929 03:25:09.486627 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10478
I20260929 03:25:09.490936 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10611
2026-09-29T03:25:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18334 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:25:09 dist-test-slave-fzwk krb5kdc[10798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:25:09 dist-test-slave-fzwk krb5kdc[10798](info): set up 2 sockets
Sep 29 03:25:09 dist-test-slave-fzwk krb5kdc[10798](info): commencing operation
krb5kdc: starting...
W20260929 03:25:10.539654 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:25:10 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:25:10.657888 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:10.740907 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:10.741192 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:10.741230 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:10.742476 10811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:25:10.742561 10811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:25:10.742583 10811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:25:10.742601 10811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:25:10.742633 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:10.742664 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:10.742686 10811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:25:10.742718 10811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:25:10.744383 10811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:10.744704 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:10.744948 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:10.749970 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:10.750130 10811 server_base.cc:1034] running on GCE node
W20260929 03:25:10.749963 10816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:10.750042 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:10.750522 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:10.750819 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:10.751961 10811 hybrid_clock.cc:648] HybridClock initialized: now 1790652310751937 us; error 43 us; skew 500 ppm
Sep 29 03:25:10 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:10.754597 10811 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:25:10.755121 10811 webserver.cc:460] Webserver started at http://127.27.68.126:33635/ using document root <none> and password file <none>
I20260929 03:25:10.755400 10811 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:10.755450 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:10.755622 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:25:10.756762 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "fe2270a5a41e4ede9786428fd1daf35d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "15ab2ae9c68c3442d862dd268b3e216b"
server_key_iv: "e767136c9281e9c9fd1dace9f734f64c"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.757144 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "fe2270a5a41e4ede9786428fd1daf35d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "15ab2ae9c68c3442d862dd268b3e216b"
server_key_iv: "e767136c9281e9c9fd1dace9f734f64c"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.758594 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:25:10.759436 10826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:10.759634 10811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:10.759732 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fe2270a5a41e4ede9786428fd1daf35d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "15ab2ae9c68c3442d862dd268b3e216b"
server_key_iv: "e767136c9281e9c9fd1dace9f734f64c"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.759809 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:10.779999 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:10.782058 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:10.782217 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:10.786594 10811 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35575
I20260929 03:25:10.786651 10878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35575 every 8 connection(s)
I20260929 03:25:10.787053 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:25:10.787971 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:10.790307 10879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap starting.
I20260929 03:25:10.790899 10879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:10.791152 10879 log.cc:824] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:10.791911 10879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: No bootstrap required, opened a new log
I20260929 03:25:10.792886 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10811
I20260929 03:25:10.792999 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:25:10.793210 27921 external_mini_cluster.cc:1428] Setting key 3f8100c3eca61e68f248f70ca1140b41
I20260929 03:25:10.794183 10879 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:10.794452 10879 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:10.794518 10879 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Initialized, Role: FOLLOWER
I20260929 03:25:10.794670 10879 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:10.794765 10879 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:25:10.794801 10879 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:25:10.794905 10879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:10.795996 10879 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:10.796128 10879 leader_election.cc:302] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2270a5a41e4ede9786428fd1daf35d; no voters:
I20260929 03:25:10.796305 10879 leader_election.cc:288] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:25:10.796352 10884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:25:10.796509 10884 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 LEADER]: Becoming Leader. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Running, Role: LEADER
I20260929 03:25:10.796602 10879 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:25:10.796619 10884 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:10.796940 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2270a5a41e4ede9786428fd1daf35d. Latest consensus state: current_term: 1 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:10.797010 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
I20260929 03:25:10.796977 10886 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:10.797214 10886 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
W20260929 03:25:10.798466 10900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:25:10.798527 10900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:25:10.798815 10890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:25:10.799027 10890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:25:10 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:10.801110 10890 catalog_manager.cc:1347] Generated new cluster ID: 034e9641e5134fdda74d20a3ef5672ce
I20260929 03:25:10.801175 10890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:25:10.805711 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:10.793842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.27.68.126:35575)
0929 03:25:10.794062 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:10.794077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:10.794105 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:25:10.794209 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:10.794215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:10.794260 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:10.794400 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:10.795242 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.796285 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.797401 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.797989 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.798476 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:10.801107 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:10.801112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:10.801112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:10.801126 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:10.803275 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:10.803769 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:10.803773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:10.803773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:10.803908 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:10.804133 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:10.804134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:10.804134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:10.804482 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:10.804632 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:10.805294 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 03:25:10.805459 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260929 03:25:10.820947 10890 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:25:10.821743 10890 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:25:10.826586 10890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Generated new TSK 0
I20260929 03:25:10.826812 10890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:25:10.854123 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:10.939978 10905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:10.940224 10905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:10.940253 10905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:10.941408 10905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:10.941455 10905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:10.941483 10905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:25:10.942926 10905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:10.943140 10905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:10.943439 10905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:10.948645 10913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:10.948645 10911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:10.948642 10910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:10.948783 10905 server_base.cc:1034] running on GCE node
I20260929 03:25:10.949018 10905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:10.949324 10905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:10.950508 10905 hybrid_clock.cc:648] HybridClock initialized: now 1790652310950496 us; error 35 us; skew 500 ppm
Sep 29 03:25:10 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:10.953274 10905 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:25:10.953838 10905 webserver.cc:460] Webserver started at http://127.27.68.65:36437/ using document root <none> and password file <none>
I20260929 03:25:10.954088 10905 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:10.954161 10905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:10.954337 10905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:25:10.955569 10905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "b203fee6638d4440bcc1d5f10623545d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "5aa5a7fd1b1df23d9aef8f4a8ba80a37"
server_key_iv: "4e25dc9f10937be2a75cbe5f61b0b286"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.955971 10905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "b203fee6638d4440bcc1d5f10623545d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "5aa5a7fd1b1df23d9aef8f4a8ba80a37"
server_key_iv: "4e25dc9f10937be2a75cbe5f61b0b286"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.957482 10905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:25:10.958293 10920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:10.958508 10905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:10.958616 10905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "b203fee6638d4440bcc1d5f10623545d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "5aa5a7fd1b1df23d9aef8f4a8ba80a37"
server_key_iv: "4e25dc9f10937be2a75cbe5f61b0b286"
server_key_version: "encryptionkey@0"
I20260929 03:25:10.958702 10905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:10.972874 10905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:10.975268 10905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:10.975399 10905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:10.975600 10905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:10.975934 10905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:10.975970 10905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:10.976011 10905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:10.976037 10905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:10.981967 10905 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:41735
I20260929 03:25:10.982018 11033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:41735 every 8 connection(s)
I20260929 03:25:10.982347 10905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:25:10 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:10.989358 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 10905
I20260929 03:25:10.989442 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:25:10.989634 27921 external_mini_cluster.cc:1428] Setting key 708f8dd73137d817b0c5a560a182201d
I20260929 03:25:10.992033 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:10.983080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:47573 (local address 127.27.68.126:35575)
0929 03:25:10.983142 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:10.983144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:10.983358 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:25:10.983837 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:10.983838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:10.983851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:10.983913 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:10.984888 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.985396 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.986368 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.986570 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.986623 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:10.988797 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:10.988799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:10.988799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:10.988808 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:10.990515 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:10.990957 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:10.990959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:10.990959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:10.991027 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:10.991319 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:10.991320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:10.991320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:10.991480 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:10.991552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:10.991825 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:25:10.991941 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:25:10.992214 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:10.983040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:47573)
0929 03:25:10.983202 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:10.983261 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:10.983754 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:10.983978 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:10.983981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:10.984592 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:10.984797 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:10.984803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.985478 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.985479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:10.986291 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:10.986293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:10.986646 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:10.986646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:10.986752 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:10.987286 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:10.987297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:10.988698 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:10.990611 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:10.990618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:10.990620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:10.990873 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:10.991105 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:10.991105 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:10.991105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:10.991252 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:10.991545 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:10.991550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:10.991675 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:25:10.991785 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:25:10.991954 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260929 03:25:10.992825 11034 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:10.992933 11034 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:10.993129 11034 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:10.993662 10843 ts_manager.cc:194] Registered new tserver with Master: b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:10.994607 10843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:47573
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:25:11.015986 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:11.100577 11039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:11.100801 11039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:11.100827 11039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:11.101970 11039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:11.102010 11039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:11.102037 11039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:25:11.103968 11039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:11.104190 11039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:11.104379 11039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:11.109486 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:11.109499 11044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:11.109493 11045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:11.109766 11039 server_base.cc:1034] running on GCE node
I20260929 03:25:11.110003 11039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:11.110284 11039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:11.111456 11039 hybrid_clock.cc:648] HybridClock initialized: now 1790652311111436 us; error 32 us; skew 500 ppm
Sep 29 03:25:11 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:11.114111 11039 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:25:11.114519 11039 webserver.cc:460] Webserver started at http://127.27.68.66:36413/ using document root <none> and password file <none>
I20260929 03:25:11.114711 11039 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:11.114750 11039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:11.114837 11039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:25:11.116029 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "e08a3b642ae5459eb15363c016113b63"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "268db44f3474852fa19138fcde08a3ea"
server_key_iv: "30fddd4fdfabba20a6675228c9dd5ad4"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.116328 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "e08a3b642ae5459eb15363c016113b63"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "268db44f3474852fa19138fcde08a3ea"
server_key_iv: "30fddd4fdfabba20a6675228c9dd5ad4"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.117605 11039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:25:11.118283 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.118428 11039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:25:11.118510 11039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "e08a3b642ae5459eb15363c016113b63"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "268db44f3474852fa19138fcde08a3ea"
server_key_iv: "30fddd4fdfabba20a6675228c9dd5ad4"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.118570 11039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:11.132609 11039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:11.134488 11039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:11.134603 11039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:11.134789 11039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:11.135134 11039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:11.135169 11039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.135212 11039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:11.135268 11039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.140930 11039 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42119
I20260929 03:25:11.141037 11167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42119 every 8 connection(s)
I20260929 03:25:11.141352 11039 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:25:11 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:11.151181 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11039
I20260929 03:25:11.151336 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:25:11.151515 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.142154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:51125 (local address 127.27.68.126:35575)
0929 03:25:11.142210 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.142213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.142464 (+ 251us) server_negotiation.cc:407] Connection header received
0929 03:25:11.142942 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.142943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.142953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.143014 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:11.144043 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.144540 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.146026 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.146233 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.146309 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.148600 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:11.148602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:11.148602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:11.148611 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:11.150037 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.150469 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.150470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.150470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.150532 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.150788 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.150789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.150789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.150914 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:11.151001 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.151321 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.151419 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:25:11.151552 27921 external_mini_cluster.cc:1428] Setting key 0ca79e651e5eaf058bbb12d6f42289c0
I20260929 03:25:11.151746 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.142142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:51125)
0929 03:25:11.142335 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.142374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.142858 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.143100 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.143103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.143756 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:11.143962 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.143966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.144646 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.144647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.145932 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.145935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.146328 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.146329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.146476 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.147034 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:11.147043 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:11.148474 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:11.150137 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:11.150147 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:11.150149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:11.150382 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:11.150603 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:11.150604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:11.150604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:11.150723 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:11.150995 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:11.151001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:11.151164 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.151300 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.151480 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260929 03:25:11.152245 11168 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:11.152360 11168 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:11.152549 11168 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:11.152951 10843 ts_manager.cc:194] Registered new tserver with Master: e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
I20260929 03:25:11.153468 10843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:51125
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:25:11.178459 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:11.264004 11173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:11.264210 11173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:11.264235 11173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:11.265434 11173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:11.265478 11173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:11.265504 11173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:25:11.266956 11173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:11.267167 11173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:11.267410 11173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:11.272552 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:11.272552 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:11.272575 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:11.272591 11173 server_base.cc:1034] running on GCE node
I20260929 03:25:11.272960 11173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:11.273247 11173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:11.274425 11173 hybrid_clock.cc:648] HybridClock initialized: now 1790652311274408 us; error 45 us; skew 500 ppm
Sep 29 03:25:11 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:11.277030 11173 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:25:11.277500 11173 webserver.cc:460] Webserver started at http://127.27.68.67:45843/ using document root <none> and password file <none>
I20260929 03:25:11.277740 11173 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:11.277807 11173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:11.277961 11173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:25:11.279054 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "c67992b798e04feaa5ea20c375cbb5bd"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "2370b7d8ead4c90bbae79dbe5e3b358f"
server_key_iv: "42d5521d904c33057703d6c6c8db7747"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.279464 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "c67992b798e04feaa5ea20c375cbb5bd"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "2370b7d8ead4c90bbae79dbe5e3b358f"
server_key_iv: "42d5521d904c33057703d6c6c8db7747"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.280807 11173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:11.281522 11188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.281687 11173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:25:11.281790 11173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "c67992b798e04feaa5ea20c375cbb5bd"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "2370b7d8ead4c90bbae79dbe5e3b358f"
server_key_iv: "42d5521d904c33057703d6c6c8db7747"
server_key_version: "encryptionkey@0"
I20260929 03:25:11.281873 11173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:11.299484 11173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:11.301798 11173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:11.301918 11173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:11.302199 11173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:11.302486 11173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:11.302518 11173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.302542 11173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:11.302556 11173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:11.308419 11173 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:43495
I20260929 03:25:11.308503 11301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:43495 every 8 connection(s)
I20260929 03:25:11.308811 11173 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:25:11.314626 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11173
I20260929 03:25:11.314728 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:25:11.314922 27921 external_mini_cluster.cc:1428] Setting key 095a9df2c0fee32190cdb79474111fa5
Sep 29 03:25:11 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:11.318648 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.309596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:42021 (local address 127.27.68.126:35575)
0929 03:25:11.309647 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.309650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.309876 (+ 226us) server_negotiation.cc:407] Connection header received
0929 03:25:11.310357 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.310358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.310368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.310421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:11.311488 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.311970 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.312996 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.313191 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.313258 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.315680 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:11.315682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:11.315682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:11.315690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:11.317244 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.317667 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.317668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.317668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.317724 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.317987 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.317988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.317988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.318109 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:11.318197 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.318473 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.318554 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:25:11.318856 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.309593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:42021)
0929 03:25:11.309752 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.309791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.310272 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.310476 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.310479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.311127 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:11.311400 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.311407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.312048 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.312048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.312885 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.312887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.313271 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.313272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.313391 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.313884 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:11.313893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:11.315581 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:11.317345 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:11.317352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:11.317354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:11.317592 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:11.317807 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:11.317808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:11.317808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:11.317923 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:11.318188 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:11.318194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:11.318338 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.318440 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.318611 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260929 03:25:11.319327 11302 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:11.319435 11302 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:11.319622 11302 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:11.319993 10843 ts_manager.cc:194] Registered new tserver with Master: c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495)
I20260929 03:25:11.320343 10843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:42021
I20260929 03:25:11.326793 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:25:11.333990 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.328227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.27.68.126:35575)
0929 03:25:11.328295 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.328297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.328408 (+ 111us) server_negotiation.cc:407] Connection header received
0929 03:25:11.328461 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.328473 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.328482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.328522 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:11.329042 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.329520 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.330211 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.330386 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.330436 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.331657 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:11.331658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:11.331659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:11.331665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:11.333046 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.333355 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.333356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.333356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.333405 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:11.333560 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:11.333560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:11.333561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:11.333704 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:11.333780 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.333865 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.333923 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:25:11.335567 10843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:25:11.337184 10843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:25:11.343617 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.27.68.66:42119)
0929 03:25:11.340047 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.340051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.340060 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:25:11.340102 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.340104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.340118 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.340187 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:11.340452 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341503 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.342633 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343192 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.343316 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.343380 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.343499 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.343537 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":77,"threads_started":1}
I20260929 03:25:11.343609 11315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42119 (local address 127.0.0.1:34198)
0929 03:25:11.339700 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.339710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.339749 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.340322 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.340323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.340325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:11.340369 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.340370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341614 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.341615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.342552 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.342554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343318 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.343319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.343394 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.343406 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.343485 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.343535 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260929 03:25:11.343756 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43495 (local address 127.0.0.1:56814)
0929 03:25:11.339513 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.339559 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.339631 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.340129 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.340133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.340135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:11.340213 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.340215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341584 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.341585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.342642 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.342645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343540 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.343541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.343607 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.343609 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.343637 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.343671 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260929 03:25:11.343834 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.0.0.1:48792)
0929 03:25:11.339864 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.339875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.339928 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.340258 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.340259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.340261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:11.340306 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.340308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341191 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.341192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.342390 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.342393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343439 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.343440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.343510 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.343515 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.343572 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.343618 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260929 03:25:11.343894 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.27.68.65:41735)
0929 03:25:11.340021 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.340026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.340036 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:25:11.340073 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.340075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.340090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.340171 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:11.340382 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341096 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.342471 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343277 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.343347 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.343402 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.343665 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.343808 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":117,"threads_started":1}
I20260929 03:25:11.344512 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.339486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.27.68.67:43495)
0929 03:25:11.339899 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.339904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.339915 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:25:11.339956 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.339958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.339977 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.340290 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:11.340354 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.341476 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.342720 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.343455 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.344305 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.344370 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.344393 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.344434 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":94,"threads_started":1}
I20260929 03:25:11.344861 11102 tablet_service.cc:1467] Processing CreateTablet for tablet e8033855372245dfb93d6e9d29b68d5d (DEFAULT_TABLE table=test-table [id=87120552019f4132a7770a91a834baee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:11.344866 10968 tablet_service.cc:1467] Processing CreateTablet for tablet e8033855372245dfb93d6e9d29b68d5d (DEFAULT_TABLE table=test-table [id=87120552019f4132a7770a91a834baee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:11.345070 11233 tablet_service.cc:1467] Processing CreateTablet for tablet e8033855372245dfb93d6e9d29b68d5d (DEFAULT_TABLE table=test-table [id=87120552019f4132a7770a91a834baee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:11.345196 11102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8033855372245dfb93d6e9d29b68d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:11.345214 10968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8033855372245dfb93d6e9d29b68d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:11.345353 11233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8033855372245dfb93d6e9d29b68d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:11.347860 11319 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Bootstrap starting.
I20260929 03:25:11.348419 11319 tablet_bootstrap.cc:654] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:11.348656 11320 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Bootstrap starting.
I20260929 03:25:11.349030 11319 log.cc:824] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:11.349139 11321 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Bootstrap starting.
I20260929 03:25:11.349489 11320 tablet_bootstrap.cc:654] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:11.349779 11321 tablet_bootstrap.cc:654] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:11.349885 11319 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: No bootstrap required, opened a new log
I20260929 03:25:11.349967 11319 ts_tablet_manager.cc:1397] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:25:11.350041 11320 log.cc:824] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:11.350350 11321 log.cc:824] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:11.350903 11320 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: No bootstrap required, opened a new log
I20260929 03:25:11.350970 11320 ts_tablet_manager.cc:1397] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:25:11.351110 11321 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: No bootstrap required, opened a new log
I20260929 03:25:11.351161 11321 ts_tablet_manager.cc:1397] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:25:11.351823 11319 raft_consensus.cc:348] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.351994 11319 raft_consensus.cc:374] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:11.352036 11319 raft_consensus.cc:729] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a3b642ae5459eb15363c016113b63, State: Initialized, Role: FOLLOWER
I20260929 03:25:11.352176 11319 consensus_queue.cc:260] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.352468 11168 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
I20260929 03:25:11.352474 11319 ts_tablet_manager.cc:1428] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:25:11.352964 11320 raft_consensus.cc:348] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.353139 11320 raft_consensus.cc:374] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:11.353178 11320 raft_consensus.cc:729] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67992b798e04feaa5ea20c375cbb5bd, State: Initialized, Role: FOLLOWER
I20260929 03:25:11.353102 11321 raft_consensus.cc:348] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.353257 11321 raft_consensus.cc:374] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:11.353304 11321 raft_consensus.cc:729] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b203fee6638d4440bcc1d5f10623545d, State: Initialized, Role: FOLLOWER
I20260929 03:25:11.353297 11320 consensus_queue.cc:260] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.353400 11321 consensus_queue.cc:260] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.353580 11320 ts_tablet_manager.cc:1428] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:25:11.353621 11302 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
I20260929 03:25:11.353652 11321 ts_tablet_manager.cc:1428] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:25:11.353780 11034 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
W20260929 03:25:11.392023 11169 tablet.cc:2369] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:25:11.393973 11325 raft_consensus.cc:482] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:11.394088 11325 raft_consensus.cc:504] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.394502 11325 leader_election.cc:288] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:11.397627 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.394714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43495 (local address 127.27.68.66:40881)
0929 03:25:11.394778 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.394793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.394857 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.395077 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.395078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.395080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:11.395155 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.395157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.395833 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.395834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.396697 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.396699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.397341 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.397342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.397431 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.397435 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.397500 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.397551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:25:11.397657 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.394812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:40881 (local address 127.27.68.67:43495)
0929 03:25:11.394860 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.394862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.394873 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:25:11.394997 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.394998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.395006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.395074 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:11.395253 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.395745 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.396785 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.397263 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.397344 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.397354 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.397551 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.397594 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:25:11.397742 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.394907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:55663 (local address 127.27.68.65:41735)
0929 03:25:11.394961 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.394963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.395088 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:25:11.395163 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.395164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.395173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.395278 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:11.395440 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.395914 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.396933 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.397384 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.397464 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.397473 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.397642 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.397682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:25:11.397748 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.394827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.27.68.66:55663)
0929 03:25:11.395002 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.395013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:11.395102 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:11.395285 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:11.395286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:11.395287 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:11.395359 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.395361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.395999 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.396000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.396830 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:11.396832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.397462 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.397463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:11.397552 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.397556 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:25:11.397624 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:25:11.397664 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":128,"threads_started":1}
I20260929 03:25:11.398275 11256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8033855372245dfb93d6e9d29b68d5d" candidate_uuid: "e08a3b642ae5459eb15363c016113b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67992b798e04feaa5ea20c375cbb5bd" is_pre_election: true
I20260929 03:25:11.398274 10988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8033855372245dfb93d6e9d29b68d5d" candidate_uuid: "e08a3b642ae5459eb15363c016113b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b203fee6638d4440bcc1d5f10623545d" is_pre_election: true
I20260929 03:25:11.398455 11256 raft_consensus.cc:2454] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a3b642ae5459eb15363c016113b63 in term 0.
I20260929 03:25:11.398455 10988 raft_consensus.cc:2454] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08a3b642ae5459eb15363c016113b63 in term 0.
I20260929 03:25:11.398694 11058 leader_election.cc:302] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d, e08a3b642ae5459eb15363c016113b63; no voters:
I20260929 03:25:11.398828 11325 raft_consensus.cc:2781] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:25:11.398888 11325 raft_consensus.cc:482] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:25:11.398913 11325 raft_consensus.cc:3037] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:11.399729 11325 raft_consensus.cc:504] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.399873 11325 leader_election.cc:288] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [CANDIDATE]: Term 1 election: Requested vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:11.400035 11256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8033855372245dfb93d6e9d29b68d5d" candidate_uuid: "e08a3b642ae5459eb15363c016113b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67992b798e04feaa5ea20c375cbb5bd"
I20260929 03:25:11.400099 10988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8033855372245dfb93d6e9d29b68d5d" candidate_uuid: "e08a3b642ae5459eb15363c016113b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b203fee6638d4440bcc1d5f10623545d"
I20260929 03:25:11.400138 11256 raft_consensus.cc:3037] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:11.400187 10988 raft_consensus.cc:3037] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:11.400915 10988 raft_consensus.cc:2454] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a3b642ae5459eb15363c016113b63 in term 1.
I20260929 03:25:11.400918 11256 raft_consensus.cc:2454] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08a3b642ae5459eb15363c016113b63 in term 1.
I20260929 03:25:11.401121 11058 leader_election.cc:302] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63; no voters:
I20260929 03:25:11.401228 11325 raft_consensus.cc:2781] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:25:11.401409 11325 raft_consensus.cc:686] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 LEADER]: Becoming Leader. State: Replica: e08a3b642ae5459eb15363c016113b63, State: Running, Role: LEADER
I20260929 03:25:11.401512 11325 consensus_queue.cc:237] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:11.402338 10843 catalog_manager.cc:5495] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to e08a3b642ae5459eb15363c016113b63 (127.27.68.66). New cstate: current_term: 1 leader_uuid: "e08a3b642ae5459eb15363c016113b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260929 03:25:11.420588 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.418041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.27.68.66:42119)
0929 03:25:11.418104 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:11.418106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:11.418116 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:25:11.418190 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:11.418191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:11.418212 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:11.418302 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:25:11.418442 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.418920 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.419763 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:11.419982 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:11.420055 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:11.420221 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:25:11.420339 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:25:11.420431 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:11.420451 (+ 20us) server_negotiation.cc:299] Negotiation successful
0929 03:25:11.420475 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:25:11.422405 11256 raft_consensus.cc:1261] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Refusing update from remote peer e08a3b642ae5459eb15363c016113b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:25:11.422403 10988 raft_consensus.cc:1261] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Refusing update from remote peer e08a3b642ae5459eb15363c016113b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:25:11.422811 11329 consensus_queue.cc:1035] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:25:11.422950 11325 consensus_queue.cc:1035] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:25:11.425274 11332 mvcc.cc:205] Tried to move back new op lower bound from 7334511867583365120 to 7334511867501174784. Current Snapshot: MvccSnapshot[applied={T|T < 7334511867583365120}]
I20260929 03:25:11.426126 11333 mvcc.cc:205] Tried to move back new op lower bound from 7334511867583365120 to 7334511867501174784. Current Snapshot: MvccSnapshot[applied={T|T < 7334511867583365120}]
I20260929 03:25:11.426144 11334 mvcc.cc:205] Tried to move back new op lower bound from 7334511867583365120 to 7334511867501174784. Current Snapshot: MvccSnapshot[applied={T|T < 7334511867583365120}]
I20260929 03:25:11.427827 10842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:25:11.427942 10842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:25:11.428728 10842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Sending DeleteTablet for 3 replicas of tablet e8033855372245dfb93d6e9d29b68d5d
I20260929 03:25:11.429106 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10905
I20260929 03:25:11.429142 10968 tablet_service.cc:1514] Processing DeleteTablet for tablet e8033855372245dfb93d6e9d29b68d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:48792
I20260929 03:25:11.429142 11233 tablet_service.cc:1514] Processing DeleteTablet for tablet e8033855372245dfb93d6e9d29b68d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:56814
I20260929 03:25:11.429339 11102 tablet_service.cc:1514] Processing DeleteTablet for tablet e8033855372245dfb93d6e9d29b68d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:34198
I20260929 03:25:11.431034 11341 tablet_replica.cc:330] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: stopping tablet replica
I20260929 03:25:11.431164 11341 raft_consensus.cc:2229] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:25:11.431305 11341 raft_consensus.cc:2258] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:11.431802 11341 ts_tablet_manager.cc:1905] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:25:11.434047 11341 ts_tablet_manager.cc:1918] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:25:11.434131 11341 log.cc:1196] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/e8033855372245dfb93d6e9d29b68d5d
I20260929 03:25:11.434418 11341 ts_tablet_manager.cc:1939] T e8033855372245dfb93d6e9d29b68d5d P c67992b798e04feaa5ea20c375cbb5bd: Deleting consensus metadata
I20260929 03:25:11.434887 10830 catalog_manager.cc:4841] TS c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): tablet e8033855372245dfb93d6e9d29b68d5d (table test-table [id=87120552019f4132a7770a91a834baee]) successfully deleted
W20260929 03:25:11.435307 10830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:41735 (error 108)
I20260929 03:25:11.435549 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11039
I20260929 03:25:11.435570 11342 tablet_replica.cc:330] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: stopping tablet replica
I20260929 03:25:11.435676 11342 raft_consensus.cc:2229] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:25:11.435808 11342 raft_consensus.cc:2258] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:11.435876 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.435756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.0.0.1:48798)
0929 03:25:11.435808 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.435842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:25:11.435976 10830 catalog_manager.cc:4584] TS b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8033855372245dfb93d6e9d29b68d5d: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41735: connect: Connection refused (error 111)
I20260929 03:25:11.440954 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:11.440844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42119 (local address 127.0.0.1:34212)
0929 03:25:11.440894 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:11.440922 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.66:42119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:25:11.441056 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11173
I20260929 03:25:11.446332 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 10811
I20260929 03:25:31.452003 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--webserver_interface=127.27.68.126
--webserver_port=33635
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:31.534344 11344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:31.534615 11344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:31.534657 11344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:31.535949 11344 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:25:31.536019 11344 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:25:31.536041 11344 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:25:31.536060 11344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:25:31.536088 11344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:31.536123 11344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:31.536146 11344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:25:31.536177 11344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:25:31.537729 11344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=33635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:31.538048 11344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:31.538291 11344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:31.546706 11353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.546706 11350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.546710 11351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.547060 11344 server_base.cc:1034] running on GCE node
I20260929 03:25:31.547390 11344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:31.547730 11344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:31.548904 11344 hybrid_clock.cc:648] HybridClock initialized: now 1790652331548892 us; error 31 us; skew 500 ppm
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:31.555779 11344 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:25:31.556304 11344 webserver.cc:460] Webserver started at http://127.27.68.126:33635/ using document root <none> and password file <none>
I20260929 03:25:31.556522 11344 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:31.556577 11344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:31.558063 11344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:31.558763 11360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:31.558967 11344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:31.559067 11344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fe2270a5a41e4ede9786428fd1daf35d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "15ab2ae9c68c3442d862dd268b3e216b"
server_key_iv: "e767136c9281e9c9fd1dace9f734f64c"
server_key_version: "encryptionkey@0"
I20260929 03:25:31.559561 11344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:31.584981 11344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:31.587057 11344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:31.587226 11344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:31.591323 11344 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35575
I20260929 03:25:31.591396 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35575 every 8 connection(s)
I20260929 03:25:31.591825 11344 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:25:31.593403 11413 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:25:31.594020 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap starting.
I20260929 03:25:31.597491 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11344
I20260929 03:25:31.597834 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:41735
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=36437
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:25:31.603281 11413 log.cc:824] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:31.605331 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:25:31.605693 11413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap complete.
I20260929 03:25:31.608119 11413 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:31.608352 11413 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Initialized, Role: FOLLOWER
I20260929 03:25:31.608536 11413 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:31.608604 11413 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:25:31.608659 11413 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:25:31.608709 11413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:25:31.609797 11413 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:31.609889 11413 leader_election.cc:302] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2270a5a41e4ede9786428fd1daf35d; no voters:
I20260929 03:25:31.610129 11413 leader_election.cc:288] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:25:31.610167 11417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:25:31.610287 11417 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 LEADER]: Becoming Leader. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Running, Role: LEADER
I20260929 03:25:31.610430 11413 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:25:31.610585 11417 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:31.611019 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2270a5a41e4ede9786428fd1daf35d. Latest consensus state: current_term: 2 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:31.611112 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
I20260929 03:25:31.611264 11419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:31.611318 11419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
I20260929 03:25:31.611778 11430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:25:31.612739 11430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:25:31.612870 11430 catalog_manager.cc:1259] Loaded cluster ID: 034e9641e5134fdda74d20a3ef5672ce
I20260929 03:25:31.612926 11430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:25:31.613729 11430 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:25:31.613914 11430 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Loaded TSK: 0
I20260929 03:25:31.625022 11430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Generated new TSK 1
I20260929 03:25:31.625149 11430 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:25:31.700495 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:31.700760 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:31.700803 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:31.702010 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:31.702085 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:31.702131 11415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:25:31.703754 11415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:41735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=36437
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:31.704031 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:31.704279 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:31.709383 11442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.709370 11441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.709651 11415 server_base.cc:1034] running on GCE node
W20260929 03:25:31.709367 11444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.709968 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:31.710264 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:31.711403 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790652331711383 us; error 38 us; skew 500 ppm
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:31.714044 11415 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:25:31.714561 11415 webserver.cc:460] Webserver started at http://127.27.68.65:36437/ using document root <none> and password file <none>
I20260929 03:25:31.714809 11415 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:31.714891 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:31.716403 11415 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:31.717149 11451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:31.717329 11415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:31.717417 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "b203fee6638d4440bcc1d5f10623545d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "5aa5a7fd1b1df23d9aef8f4a8ba80a37"
server_key_iv: "4e25dc9f10937be2a75cbe5f61b0b286"
server_key_version: "encryptionkey@0"
I20260929 03:25:31.717762 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:31.732779 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:31.735140 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:31.735334 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:31.735581 11415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:31.736094 11458 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:25:31.737543 11415 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:25:31.737596 11415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:25:31.737632 11415 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:25:31.738306 11415 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:25:31.738350 11415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:31.738399 11458 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Bootstrap starting.
I20260929 03:25:31.746168 11415 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:41735
I20260929 03:25:31.746351 11565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:41735 every 8 connection(s)
I20260929 03:25:31.746713 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:25:31.750226 11458 log.cc:824] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:31.753854 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11415
I20260929 03:25:31.753876 11458 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:25:31.754172 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:42119
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=36413
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:25:31.754211 11458 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Bootstrap complete.
I20260929 03:25:31.755533 11458 ts_tablet_manager.cc:1397] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:31.756908 11458 raft_consensus.cc:348] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:31.757251 11458 raft_consensus.cc:729] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b203fee6638d4440bcc1d5f10623545d, State: Initialized, Role: FOLLOWER
I20260929 03:25:31.757431 11458 consensus_queue.cc:260] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:31.757704 11458 ts_tablet_manager.cc:1428] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:25:31.762422 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.747477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:38493)
0929 03:25:31.747744 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:31.747785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:31.748371 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:31.749027 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:31.749033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:31.749532 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:31.749847 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.749856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.751614 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.751615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.752451 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.752454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.753138 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.753139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.753282 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.755576 (+ 2294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:31.755590 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:31.757563 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:31.760385 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.760393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.760395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.760714 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.761050 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.761051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.761051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.761181 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.761689 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:31.761695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:31.761846 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:25:31.761958 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:25:31.762156 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260929 03:25:31.762498 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.747605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:38493 (local address 127.27.68.126:35575)
0929 03:25:31.748585 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:31.748595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:31.748632 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:25:31.748758 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:31.748763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:31.748815 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:31.748950 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:31.750003 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.751465 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.752577 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.753009 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.753148 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.757695 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:31.757700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:31.757701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:31.757715 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:31.760271 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.760819 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.760823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.760824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.760965 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.761256 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.761257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.761257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.761609 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:31.761709 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:31.762038 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 03:25:31.762256 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":106,"threads_started":1}
I20260929 03:25:31.763311 11566 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:31.763418 11566 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:31.763650 11566 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:31.764498 11377 ts_manager.cc:194] Registered new tserver with Master: b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:31.765589 11377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:38493
I20260929 03:25:31.766517 11566 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
I20260929 03:25:31.770309 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.765106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.0.0.1:41154)
0929 03:25:31.767324 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:31.767356 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:31.767446 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:31.767620 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:31.767622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:31.767624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:31.767692 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.767694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.768523 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.768524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.769359 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.769361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.770006 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.770007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.770092 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.770104 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:25:31.770179 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:25:31.770234 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":98,"threads_started":1}
I20260929 03:25:31.770310 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.765205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.27.68.65:41735)
0929 03:25:31.765361 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:31.765365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:31.767422 (+ 2057us) server_negotiation.cc:407] Connection header received
0929 03:25:31.767524 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:31.767527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:31.767542 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:31.767617 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:31.767761 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.768423 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.769450 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.769920 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.770013 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.770075 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:31.770193 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 03:25:31.770235 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260929 03:25:31.770866 11500 tablet_service.cc:1514] Processing DeleteTablet for tablet e8033855372245dfb93d6e9d29b68d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:41154
I20260929 03:25:31.771416 11575 tablet_replica.cc:330] stopping tablet replica
I20260929 03:25:31.771538 11575 raft_consensus.cc:2229] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:25:31.771611 11575 raft_consensus.cc:2258] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:31.771921 11575 ts_tablet_manager.cc:1905] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:25:31.774021 11575 ts_tablet_manager.cc:1918] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:25:31.774091 11575 log.cc:1196] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal/wals/e8033855372245dfb93d6e9d29b68d5d
I20260929 03:25:31.774346 11575 ts_tablet_manager.cc:1939] T e8033855372245dfb93d6e9d29b68d5d P b203fee6638d4440bcc1d5f10623545d: Deleting consensus metadata
I20260929 03:25:31.774915 11364 catalog_manager.cc:4841] TS b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735): tablet e8033855372245dfb93d6e9d29b68d5d (table test-table [id=87120552019f4132a7770a91a834baee]) successfully deleted
W20260929 03:25:31.843287 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:31.843526 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:31.843554 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:31.844725 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:31.844764 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:31.844791 11571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:25:31.846218 11571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:42119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=36413
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:31.846424 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:31.846617 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:31.851558 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.851611 11580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.851574 11581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.851768 11571 server_base.cc:1034] running on GCE node
I20260929 03:25:31.852006 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:31.852293 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:31.853484 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790652331853469 us; error 45 us; skew 500 ppm
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:31.856112 11571 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:25:31.856666 11571 webserver.cc:460] Webserver started at http://127.27.68.66:36413/ using document root <none> and password file <none>
I20260929 03:25:31.856914 11571 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:31.857007 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:31.858426 11571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:25:31.859129 11590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:31.859359 11571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:31.859483 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "e08a3b642ae5459eb15363c016113b63"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "268db44f3474852fa19138fcde08a3ea"
server_key_iv: "30fddd4fdfabba20a6675228c9dd5ad4"
server_key_version: "encryptionkey@0"
I20260929 03:25:31.859858 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:31.869324 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:31.871752 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:31.871925 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:31.872179 11571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:31.872617 11597 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:25:31.874101 11571 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:25:31.874164 11571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:25:31.874231 11571 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:25:31.874881 11571 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:25:31.874938 11571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:31.875005 11597 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Bootstrap starting.
I20260929 03:25:31.880934 11571 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42119
I20260929 03:25:31.881040 11704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42119 every 8 connection(s)
I20260929 03:25:31.881390 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:25:31.886709 11597 log.cc:824] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:31.889328 11597 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:25:31.889644 11597 tablet_bootstrap.cc:492] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Bootstrap complete.
I20260929 03:25:31.890555 11597 ts_tablet_manager.cc:1397] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260929 03:25:31.890796 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11571
I20260929 03:25:31.891078 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:43495
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=45843
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:25:31.892320 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.882166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:48167 (local address 127.27.68.126:35575)
0929 03:25:31.882223 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:31.882225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:31.882409 (+ 184us) server_negotiation.cc:407] Connection header received
0929 03:25:31.883177 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:31.883178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:31.883190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:31.883302 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:31.884263 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.884758 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.886462 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.886661 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.886712 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.889216 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:31.889218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:31.889219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:31.889233 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:31.890767 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.891329 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.891330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.891331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.891391 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.891636 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.891637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.891638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.891760 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:31.891844 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:31.892131 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 03:25:31.892228 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:25:31.892349 11597 raft_consensus.cc:348] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:31.892751 11597 raft_consensus.cc:729] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a3b642ae5459eb15363c016113b63, State: Initialized, Role: FOLLOWER
I20260929 03:25:31.892905 11597 consensus_queue.cc:260] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:31.893215 11597 ts_tablet_manager.cc:1428] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
I20260929 03:25:31.893446 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.882095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:48167)
0929 03:25:31.882274 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:31.882327 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:31.883051 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:31.883389 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:31.883394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:31.883796 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:31.884143 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.884153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.884895 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.884896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.886366 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.886369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.886764 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.886765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.886918 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.887524 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:31.887534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:31.889108 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:31.890886 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.890894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.890896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.891207 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.891464 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.891465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.891465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.891567 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.891834 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:31.891840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:31.891986 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 03:25:31.892109 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 03:25:31.892276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260929 03:25:31.893901 11705 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:31.894013 11705 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:31.894213 11705 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:31.894798 11377 ts_manager.cc:194] Registered new tserver with Master: e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
I20260929 03:25:31.895426 11377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:48167
I20260929 03:25:31.897050 11705 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.66@KRBTEST.COM
I20260929 03:25:31.903544 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.895252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.27.68.66:42119)
0929 03:25:31.895426 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:31.895431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:31.895443 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:25:31.895517 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:31.895521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:31.895541 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:31.895613 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:31.896200 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.896854 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.897709 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.898016 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.898107 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.900617 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:31.900625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:31.900625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:31.900632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:31.902227 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.902738 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.902740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.902741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.902793 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:31.902991 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:31.902991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:31.902992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:31.903130 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:31.903192 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:31.903396 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:25:31.903477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260929 03:25:31.903573 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:31.895157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42119 (local address 127.0.0.1:53258)
0929 03:25:31.895200 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:31.895234 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:31.895277 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:31.895616 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:31.895618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:31.896059 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:31.896114 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.896117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.896948 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.896950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.897638 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:31.897641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:31.898110 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:31.898111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:31.898171 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:31.898779 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:31.898783 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:31.900524 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:31.902320 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.902329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.902332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.902644 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.902857 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:31.902858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:31.902859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:31.902928 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:31.903194 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:31.903202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:31.903283 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 03:25:31.903360 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:25:31.903472 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:25:31.903997 11639 tablet_service.cc:1514] Processing DeleteTablet for tablet e8033855372245dfb93d6e9d29b68d5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:11 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53258
I20260929 03:25:31.904632 11712 tablet_replica.cc:330] stopping tablet replica
I20260929 03:25:31.904747 11712 raft_consensus.cc:2229] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:25:31.904837 11712 raft_consensus.cc:2258] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:31.905151 11712 ts_tablet_manager.cc:1905] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:25:31.907603 11712 ts_tablet_manager.cc:1918] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:25:31.907696 11712 log.cc:1196] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/e8033855372245dfb93d6e9d29b68d5d
I20260929 03:25:31.907940 11712 ts_tablet_manager.cc:1939] T e8033855372245dfb93d6e9d29b68d5d P e08a3b642ae5459eb15363c016113b63: Deleting consensus metadata
I20260929 03:25:31.908448 11364 catalog_manager.cc:4841] TS e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): tablet e8033855372245dfb93d6e9d29b68d5d (table test-table [id=87120552019f4132a7770a91a834baee]) successfully deleted
W20260929 03:25:31.979250 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:31.979470 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:31.979493 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:31.980597 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:31.980633 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:31.980659 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:25:31.982044 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:43495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=45843
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:31.982244 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:31.982431 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:31.987488 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:31.987560 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.987542 11709 server_base.cc:1034] running on GCE node
W20260929 03:25:31.987506 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:31.987838 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:31.988092 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:31.989266 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790652331989259 us; error 41 us; skew 500 ppm
Sep 29 03:25:31 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:31.991752 11709 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:25:31.992252 11709 webserver.cc:460] Webserver started at http://127.27.68.67:45843/ using document root <none> and password file <none>
I20260929 03:25:31.992496 11709 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:31.992578 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:31.993939 11709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:31.994530 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:31.994701 11709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:25:31.994799 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "c67992b798e04feaa5ea20c375cbb5bd"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "2370b7d8ead4c90bbae79dbe5e3b358f"
server_key_iv: "42d5521d904c33057703d6c6c8db7747"
server_key_version: "encryptionkey@0"
I20260929 03:25:31.995174 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:32.042557 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:32.044570 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:32.044754 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:32.045019 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:32.045357 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:32.045415 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:32.045459 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:32.045507 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:32.051725 11709 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:43495
I20260929 03:25:32.051765 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:43495 every 8 connection(s)
I20260929 03:25:32.052150 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:25:32.057592 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11709
Sep 29 03:25:32 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:25:32.062184 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.052897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:33535 (local address 127.27.68.126:35575)
0929 03:25:32.052943 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.052945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.053209 (+ 264us) server_negotiation.cc:407] Connection header received
0929 03:25:32.053747 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.053748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.053760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.053813 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:32.054807 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.055366 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.056394 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.056607 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.056692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.059248 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:32.059250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:32.059250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:32.059257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:32.060715 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.061163 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.061165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.061165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.061224 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.061490 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.061491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.061491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.061607 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:32.061682 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.061998 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.062091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:25:32.062397 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.052897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:33535)
0929 03:25:32.053086 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:32.053122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:32.053640 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:32.053902 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:32.053906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:32.054499 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:32.054701 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.054706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.055463 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.055464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.056285 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.056287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.056701 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.056702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.056844 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.057485 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:32.057497 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:32.059125 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:32.060822 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:32.060830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:32.060832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:32.061071 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:32.061296 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:32.061297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:32.061298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:32.061416 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:32.061674 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:32.061681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:32.061833 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:25:32.061966 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:25:32.062150 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260929 03:25:32.063010 11841 heartbeater.cc:347] Connected to a master server at 127.27.68.126:35575
I20260929 03:25:32.063126 11841 heartbeater.cc:464] Registering TS with master...
I20260929 03:25:32.063365 11841 heartbeater.cc:511] Master 127.27.68.126:35575 requested a full tablet report, sending...
I20260929 03:25:32.063792 11377 ts_manager.cc:194] Registered new tserver with Master: c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495)
I20260929 03:25:32.064252 11377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:33535
I20260929 03:25:32.064821 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.058055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.27.68.126:35575)
0929 03:25:32.058211 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.058214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.058308 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:25:32.058416 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.058417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.058431 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.058509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:32.059249 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.059733 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.060516 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.060719 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.060795 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.061996 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:32.061997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:32.061998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:32.062005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:32.063687 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.064102 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.064103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.064103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.064162 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.064331 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.064332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.064332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.064446 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:32.064604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.064645 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.064729 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260929 03:25:32.065573 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:25:32.068161 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.065874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.27.68.126:35575)
0929 03:25:32.065908 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.065910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.066033 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:25:32.066092 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.066092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.066119 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.066172 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:25:32.066271 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.066730 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.067546 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.067734 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.067787 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.067942 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:25:32.067987 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:25:32.068113 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42752: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:25:32.068238 11845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42752: token expired
W20260929 03:25:32.069342 11308 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.68.126:35575)
I20260929 03:25:32.076128 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.069673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.27.68.126:35575)
0929 03:25:32.069719 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.069722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.069731 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:25:32.069757 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.069758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.069769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.069838 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:32.070364 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.071143 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.072226 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.072436 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.072531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.073697 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:32.073699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:32.073699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:32.073707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:32.074902 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.075318 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.075319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.075319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.075383 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:32.075595 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:32.075596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:32.075597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:32.075717 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:32.075827 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.075931 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.076020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:25:32.077350 11377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:25:32.079028 11377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:25:32.081278 11639 tablet_service.cc:1467] Processing CreateTablet for tablet c498780e935246c6b9ae03131d4150e5 (DEFAULT_TABLE table=test-table [id=460eac6d233e4f8388d64dba56a82fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:32.081297 11500 tablet_service.cc:1467] Processing CreateTablet for tablet c498780e935246c6b9ae03131d4150e5 (DEFAULT_TABLE table=test-table [id=460eac6d233e4f8388d64dba56a82fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:32.081485 11639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c498780e935246c6b9ae03131d4150e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:32.081512 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c498780e935246c6b9ae03131d4150e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:32.082950 11458 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Bootstrap starting.
I20260929 03:25:32.083292 11597 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Bootstrap starting.
I20260929 03:25:32.083338 11458 tablet_bootstrap.cc:654] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:32.083606 11597 tablet_bootstrap.cc:654] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:32.083921 11458 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: No bootstrap required, opened a new log
I20260929 03:25:32.083988 11458 ts_tablet_manager.cc:1397] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:25:32.084136 11458 raft_consensus.cc:348] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.084223 11597 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: No bootstrap required, opened a new log
I20260929 03:25:32.084218 11458 raft_consensus.cc:374] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:32.084262 11597 ts_tablet_manager.cc:1397] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:32.084276 11458 raft_consensus.cc:729] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b203fee6638d4440bcc1d5f10623545d, State: Initialized, Role: FOLLOWER
I20260929 03:25:32.084331 11458 consensus_queue.cc:260] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.084431 11458 ts_tablet_manager.cc:1428] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:32.084618 11597 raft_consensus.cc:348] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.084671 11597 raft_consensus.cc:374] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:32.084702 11597 raft_consensus.cc:729] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08a3b642ae5459eb15363c016113b63, State: Initialized, Role: FOLLOWER
I20260929 03:25:32.084740 11597 consensus_queue.cc:260] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.084833 11597 ts_tablet_manager.cc:1428] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:32.086676 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.081033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.27.68.67:43495)
0929 03:25:32.083035 (+ 2002us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.083041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.083052 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:25:32.083095 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.083098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.083117 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.083208 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:32.083372 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.084529 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.085815 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.086322 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.086384 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.086428 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.086548 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.086595 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1953,"thread_start_us":104,"threads_started":1}
I20260929 03:25:32.086679 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.080921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43495 (local address 127.0.0.1:39380)
0929 03:25:32.080979 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:32.080991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:32.081040 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:32.083201 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:32.083203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:32.083205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:32.083289 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.083291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.084613 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.084615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.085715 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.085718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.086398 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.086399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.086468 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.086473 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:25:32.086528 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:25:32.086595 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:25:32.087484 11775 tablet_service.cc:1467] Processing CreateTablet for tablet c498780e935246c6b9ae03131d4150e5 (DEFAULT_TABLE table=test-table [id=460eac6d233e4f8388d64dba56a82fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:25:32.087834 11775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c498780e935246c6b9ae03131d4150e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:25:32.090739 11850 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Bootstrap starting.
I20260929 03:25:32.091421 11850 tablet_bootstrap.cc:654] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Neither blocks nor log segments found. Creating new log.
I20260929 03:25:32.091909 11850 log.cc:824] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:32.092640 11850 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: No bootstrap required, opened a new log
I20260929 03:25:32.092697 11850 ts_tablet_manager.cc:1397] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:25:32.094218 11850 raft_consensus.cc:348] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.094391 11850 raft_consensus.cc:374] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:25:32.094416 11850 raft_consensus.cc:729] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67992b798e04feaa5ea20c375cbb5bd, State: Initialized, Role: FOLLOWER
I20260929 03:25:32.094570 11850 consensus_queue.cc:260] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.094852 11850 ts_tablet_manager.cc:1428] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:25:32.094892 11841 heartbeater.cc:503] Master 127.27.68.126:35575 was elected leader, sending a full tablet report...
W20260929 03:25:32.132055 11706 tablet.cc:2369] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:25:32.133427 11852 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:32.133538 11852 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.133839 11852 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119), b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:32.136798 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.134025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42119 (local address 127.27.68.67:36385)
0929 03:25:32.134086 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:32.134097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:32.134155 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:32.134336 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:32.134337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:32.134338 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:32.134415 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.134417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.135102 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.135103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.135917 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.135919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.136536 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.136616 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.136619 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:25:32.136679 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:25:32.136721 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:25:32.136798 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.134120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:36385 (local address 127.27.68.66:42119)
0929 03:25:32.134174 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.134176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.134187 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:25:32.134218 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.134219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.134243 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.134317 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:32.134492 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.135010 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136014 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.136436 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.136617 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.136686 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.136729 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:25:32.136893 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.134212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:39717 (local address 127.27.68.65:41735)
0929 03:25:32.134257 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.134259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.134379 (+ 120us) server_negotiation.cc:407] Connection header received
0929 03:25:32.134454 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.134455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.134463 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.134546 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:25:32.134718 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.135184 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136157 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.136531 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.136624 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.136791 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.136829 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:25:32.136909 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.134133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.27.68.67:39717)
0929 03:25:32.134318 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:32.134325 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:32.134391 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:32.134552 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:32.134553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:32.134555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:25:32.134632 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.134636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.135304 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.135305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.136063 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:32.136064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.136615 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.136616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:32.136700 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.136709 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:25:32.136783 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:25:32.136823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260929 03:25:32.137122 11659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c498780e935246c6b9ae03131d4150e5" candidate_uuid: "c67992b798e04feaa5ea20c375cbb5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a3b642ae5459eb15363c016113b63" is_pre_election: true
I20260929 03:25:32.137219 11520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c498780e935246c6b9ae03131d4150e5" candidate_uuid: "c67992b798e04feaa5ea20c375cbb5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b203fee6638d4440bcc1d5f10623545d" is_pre_election: true
I20260929 03:25:32.137311 11659 raft_consensus.cc:2454] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67992b798e04feaa5ea20c375cbb5bd in term 0.
I20260929 03:25:32.137380 11520 raft_consensus.cc:2454] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c67992b798e04feaa5ea20c375cbb5bd in term 0.
I20260929 03:25:32.137557 11731 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63; no voters:
I20260929 03:25:32.137732 11852 raft_consensus.cc:2781] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:25:32.137806 11852 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:25:32.137827 11852 raft_consensus.cc:3037] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:32.138561 11852 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.138702 11852 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [CANDIDATE]: Term 1 election: Requested vote from peers e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119), b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735)
I20260929 03:25:32.138867 11659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c498780e935246c6b9ae03131d4150e5" candidate_uuid: "c67992b798e04feaa5ea20c375cbb5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08a3b642ae5459eb15363c016113b63"
I20260929 03:25:32.138918 11520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c498780e935246c6b9ae03131d4150e5" candidate_uuid: "c67992b798e04feaa5ea20c375cbb5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b203fee6638d4440bcc1d5f10623545d"
I20260929 03:25:32.138971 11659 raft_consensus.cc:3037] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:32.139024 11520 raft_consensus.cc:3037] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:25:32.139813 11659 raft_consensus.cc:2454] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67992b798e04feaa5ea20c375cbb5bd in term 1.
I20260929 03:25:32.140010 11731 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63; no voters:
I20260929 03:25:32.140115 11852 raft_consensus.cc:2781] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:25:32.140218 11520 raft_consensus.cc:2454] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c67992b798e04feaa5ea20c375cbb5bd in term 1.
I20260929 03:25:32.140285 11852 raft_consensus.cc:686] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 1 LEADER]: Becoming Leader. State: Replica: c67992b798e04feaa5ea20c375cbb5bd, State: Running, Role: LEADER
I20260929 03:25:32.140391 11852 consensus_queue.cc:237] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:32.141202 11376 catalog_manager.cc:5495] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd reported cstate change: term changed from 0 to 1, leader changed from <none> to c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67). New cstate: current_term: 1 leader_uuid: "c67992b798e04feaa5ea20c375cbb5bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } health_report { overall_health: UNKNOWN } } }
I20260929 03:25:32.146394 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.144062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.27.68.67:43495)
0929 03:25:32.144110 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:32.144111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:32.144120 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:25:32.144176 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:32.144177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:32.144197 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:32.144281 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:25:32.144387 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.144856 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.145597 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:32.145798 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:32.145871 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:32.146037 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:25:32.146142 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:25:32.146240 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:32.146258 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 03:25:32.146280 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:25:32.148177 11520 raft_consensus.cc:1261] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Refusing update from remote peer c67992b798e04feaa5ea20c375cbb5bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:25:32.148217 11659 raft_consensus.cc:1261] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Refusing update from remote peer c67992b798e04feaa5ea20c375cbb5bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:25:32.148574 11854 consensus_queue.cc:1035] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:25:32.148736 11852 consensus_queue.cc:1035] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:25:32.150833 11859 mvcc.cc:205] Tried to move back new op lower bound from 7334511952476499968 to 7334511952447762432. Current Snapshot: MvccSnapshot[applied={T|T < 7334511952476499968}]
I20260929 03:25:32.151520 11858 mvcc.cc:205] Tried to move back new op lower bound from 7334511952476499968 to 7334511952447762432. Current Snapshot: MvccSnapshot[applied={T|T < 7334511952476499968}]
I20260929 03:25:32.153853 11377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:25:32.153947 11377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:25:32.154726 11377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Sending DeleteTablet for 3 replicas of tablet c498780e935246c6b9ae03131d4150e5
I20260929 03:25:32.155067 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11415
I20260929 03:25:32.155113 11639 tablet_service.cc:1514] Processing DeleteTablet for tablet c498780e935246c6b9ae03131d4150e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:32 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:53258
I20260929 03:25:32.155145 11775 tablet_service.cc:1514] Processing DeleteTablet for tablet c498780e935246c6b9ae03131d4150e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:25:32 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:39380
I20260929 03:25:32.155716 11712 tablet_replica.cc:330] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: stopping tablet replica
I20260929 03:25:32.155776 11712 raft_consensus.cc:2229] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:25:32.155838 11712 raft_consensus.cc:2258] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:32.156180 11712 ts_tablet_manager.cc:1905] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:25:32.157294 11867 tablet_replica.cc:330] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: stopping tablet replica
I20260929 03:25:32.157378 11867 raft_consensus.cc:2229] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:25:32.157472 11867 raft_consensus.cc:2258] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:25:32.157752 11712 ts_tablet_manager.cc:1918] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:25:32.157805 11712 log.cc:1196] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/c498780e935246c6b9ae03131d4150e5
I20260929 03:25:32.157920 11867 ts_tablet_manager.cc:1905] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:25:32.158067 11712 ts_tablet_manager.cc:1939] T c498780e935246c6b9ae03131d4150e5 P e08a3b642ae5459eb15363c016113b63: Deleting consensus metadata
I20260929 03:25:32.158412 11364 catalog_manager.cc:4841] TS e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): tablet c498780e935246c6b9ae03131d4150e5 (table test-table [id=460eac6d233e4f8388d64dba56a82fdb]) successfully deleted
I20260929 03:25:32.159560 11867 ts_tablet_manager.cc:1918] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:25:32.159632 11867 log.cc:1196] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/c498780e935246c6b9ae03131d4150e5
I20260929 03:25:32.159847 11867 ts_tablet_manager.cc:1939] T c498780e935246c6b9ae03131d4150e5 P c67992b798e04feaa5ea20c375cbb5bd: Deleting consensus metadata
I20260929 03:25:32.160409 11364 catalog_manager.cc:4841] TS c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): tablet c498780e935246c6b9ae03131d4150e5 (table test-table [id=460eac6d233e4f8388d64dba56a82fdb]) successfully deleted
W20260929 03:25:32.163198 11364 connection.cc:535] client connection to 127.27.68.65:41735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:25:32.163395 11364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:25:32.163571 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11571
I20260929 03:25:32.164669 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:32.164524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41735 (local address 127.0.0.1:41158)
0929 03:25:32.164590 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:32.164637 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 03:25:32.164880 11364 catalog_manager.cc:4584] TS b203fee6638d4440bcc1d5f10623545d (127.27.68.65:41735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c498780e935246c6b9ae03131d4150e5: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41735: connect: Connection refused (error 111)
I20260929 03:25:32.171686 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11709
I20260929 03:25:32.177085 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11344
I20260929 03:25:52.182492 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--webserver_interface=127.27.68.126
--webserver_port=33635
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:52.266573 11869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:52.266819 11869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:52.266842 11869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:52.268090 11869 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:25:52.268139 11869 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:25:52.268152 11869 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:25:52.268164 11869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:25:52.268178 11869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:52.268239 11869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:52.268256 11869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:25:52.268278 11869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:25:52.269837 11869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=33635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:52.270129 11869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:52.270398 11869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:52.275640 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.275676 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.275681 11869 server_base.cc:1034] running on GCE node
W20260929 03:25:52.275662 11877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.276072 11869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:52.276393 11869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:52.277563 11869 hybrid_clock.cc:648] HybridClock initialized: now 1790652352277557 us; error 39 us; skew 500 ppm
Sep 29 03:25:52 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:52.285880 11869 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:25:52.286386 11869 webserver.cc:460] Webserver started at http://127.27.68.126:33635/ using document root <none> and password file <none>
I20260929 03:25:52.286638 11869 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:52.286715 11869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:52.288271 11869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:52.288990 11884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.289183 11869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:52.289296 11869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "fe2270a5a41e4ede9786428fd1daf35d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "15ab2ae9c68c3442d862dd268b3e216b"
server_key_iv: "e767136c9281e9c9fd1dace9f734f64c"
server_key_version: "encryptionkey@0"
I20260929 03:25:52.289811 11869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:52.301844 11869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:52.304414 11869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:52.304603 11869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:52.308749 11869 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35575
I20260929 03:25:52.308805 11936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35575 every 8 connection(s)
I20260929 03:25:52.309221 11869 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:25:52.310779 11937 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:25:52.311497 11937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap starting.
I20260929 03:25:52.318140 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11869
I20260929 03:25:52.318511 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:41735
--local_ip_for_outbound_sockets=127.27.68.65
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=36437
--webserver_interface=127.27.68.65
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:25:52.320905 11937 log.cc:824] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:52.323508 11937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:25:52.323853 11937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: Bootstrap complete.
I20260929 03:25:52.326216 11937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:52.326474 11937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Initialized, Role: FOLLOWER
I20260929 03:25:52.326723 11937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:52.326825 11937 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:25:52.326876 11937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:25:52.326927 11937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:25:52.328215 11937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:52.328308 11937 leader_election.cc:302] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe2270a5a41e4ede9786428fd1daf35d; no voters:
I20260929 03:25:52.328565 11937 leader_election.cc:288] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:25:52.328598 11941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:25:52.328873 11941 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [term 3 LEADER]: Becoming Leader. State: Replica: fe2270a5a41e4ede9786428fd1daf35d, State: Running, Role: LEADER
I20260929 03:25:52.328950 11937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:25:52.329038 11941 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } }
I20260929 03:25:52.329402 11941 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe2270a5a41e4ede9786428fd1daf35d. Latest consensus state: current_term: 3 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:52.329406 11942 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fe2270a5a41e4ede9786428fd1daf35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe2270a5a41e4ede9786428fd1daf35d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35575 } } }
I20260929 03:25:52.329607 11941 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
I20260929 03:25:52.329622 11942 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d [sys.catalog]: This master's current role is: LEADER
I20260929 03:25:52.330850 11957 catalog_manager.cc:1259] Loaded cluster ID: 034e9641e5134fdda74d20a3ef5672ce
I20260929 03:25:52.330905 11957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: loading cluster ID for follower catalog manager: success
I20260929 03:25:52.332119 11957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe2270a5a41e4ede9786428fd1daf35d: acquiring CA information for follower catalog manager: success
F20260929 03:25:52.332545 11957 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd8240ab9ad google::LogMessage::Fail() at ??:0
@ 0x7fd8240ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd8240ab4de google::LogMessage::Flush() at ??:0
@ 0x7fd8240aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd826818104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd82c0086b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fd82c008989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fd82c008e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fd82c00b8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fd825f6e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd826dd26db start_thread at ??:0
@ 0x7fd822ba961f clone at ??:0
W20260929 03:25:52.417476 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:52.417701 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:52.417728 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:52.418987 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:52.419050 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:52.419080 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:25:52.420696 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:41735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=36437
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:52.420977 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:52.421248 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:52.426865 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.427114 11939 server_base.cc:1034] running on GCE node
W20260929 03:25:52.426880 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.426891 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.427452 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:52.427738 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:52.428941 11939 hybrid_clock.cc:648] HybridClock initialized: now 1790652352428897 us; error 31 us; skew 500 ppm
Sep 29 03:25:52 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:52.431546 11939 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:25:52.432008 11939 webserver.cc:460] Webserver started at http://127.27.68.65:36437/ using document root <none> and password file <none>
I20260929 03:25:52.432286 11939 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:52.432360 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:52.433804 11939 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:52.434638 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.434854 11939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:52.434968 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "b203fee6638d4440bcc1d5f10623545d"
format_stamp: "Formatted at 2026-09-29 03:25:10 on dist-test-slave-fzwk"
server_key: "5aa5a7fd1b1df23d9aef8f4a8ba80a37"
server_key_iv: "4e25dc9f10937be2a75cbe5f61b0b286"
server_key_version: "encryptionkey@0"
I20260929 03:25:52.435407 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:52.462272 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:52.464705 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:52.464879 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:52.465123 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:52.465607 11992 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:25:52.467701 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:25:52.467767 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:25:52.467816 11939 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:25:52.468451 11939 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:25:52.468519 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:25:52.468581 11992 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Bootstrap starting.
I20260929 03:25:52.475956 11939 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:41735
I20260929 03:25:52.476049 12099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:41735 every 8 connection(s)
I20260929 03:25:52.476460 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:25:52.480146 11992 log.cc:824] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:25:52.483009 11992 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:25:52.483410 11992 tablet_bootstrap.cc:492] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Bootstrap complete.
I20260929 03:25:52.484624 11992 ts_tablet_manager.cc:1397] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260929 03:25:52.484848 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 11939
I20260929 03:25:52.485132 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:42119
--local_ip_for_outbound_sockets=127.27.68.66
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=36413
--webserver_interface=127.27.68.66
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:25:52.486187 11992 raft_consensus.cc:348] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:52.486533 11992 raft_consensus.cc:729] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b203fee6638d4440bcc1d5f10623545d, State: Initialized, Role: FOLLOWER
I20260929 03:25:52.486747 11992 consensus_queue.cc:260] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:52.487106 11992 ts_tablet_manager.cc:1428] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260929 03:25:52.531175 12102 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:25:52.477286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:36671)
0929 03:25:52.477481 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.477524 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:52.478005 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:52.530955 (+ 52950us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
W20260929 03:25:52.531477 12100 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:35575 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:35575: Client connection negotiation failed: client connection to 127.27.68.126:35575: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:25:52.531862 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.531712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:41871)
0929 03:25:52.531783 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.531835 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 03:25:52.532279 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.532197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:50079)
0929 03:25:52.532238 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.532261 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:25:52.532404 12100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:25:52.574308 12104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:52.574540 12104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:52.574566 12104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:52.575855 12104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:52.575901 12104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:52.575949 12104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:25:52.577499 12104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:42119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=36413
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:52.577764 12104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:52.578025 12104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:52.582957 12111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.582947 12113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.582947 12110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.583284 12104 server_base.cc:1034] running on GCE node
I20260929 03:25:52.583496 12104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:52.583745 12104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:52.584903 12104 hybrid_clock.cc:648] HybridClock initialized: now 1790652352584890 us; error 29 us; skew 500 ppm
Sep 29 03:25:52 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:52.587518 12104 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:25:52.587982 12104 webserver.cc:460] Webserver started at http://127.27.68.66:36413/ using document root <none> and password file <none>
I20260929 03:25:52.588219 12104 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:52.588299 12104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:52.589684 12104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:25:52.590356 12120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.590538 12104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:25:52.590647 12104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "e08a3b642ae5459eb15363c016113b63"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "268db44f3474852fa19138fcde08a3ea"
server_key_iv: "30fddd4fdfabba20a6675228c9dd5ad4"
server_key_version: "encryptionkey@0"
I20260929 03:25:52.591008 12104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:52.617172 12104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:52.619752 12104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:52.619927 12104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:52.620175 12104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:52.620550 12104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:52.620605 12104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.620646 12104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:52.620675 12104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.626443 12104 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:42119
I20260929 03:25:52.626542 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:42119 every 8 connection(s)
I20260929 03:25:52.626907 12104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:25:52.628156 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.627727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:51899)
0929 03:25:52.627933 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.628009 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":126,"threads_started":1}
W20260929 03:25:52.628600 12234 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:35575 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:35575: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
I20260929 03:25:52.628891 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.628800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:34595)
0929 03:25:52.628848 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.628875 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:25:52.629154 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.629079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:54877)
0929 03:25:52.629123 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.629142 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:25:52.629256 12234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:25:52.630666 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12104
I20260929 03:25:52.630944 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:43495
--local_ip_for_outbound_sockets=127.27.68.67
--tserver_master_addrs=127.27.68.126:35575
--webserver_port=45843
--webserver_interface=127.27.68.67
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:25:52.712908 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:25:52.713128 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:25:52.713152 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:25:52.714345 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:25:52.714406 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:25:52.714433 12237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:25:52.715992 12237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:43495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=45843
--tserver_master_addrs=127.27.68.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:25:52.716270 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:25:52.716526 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:25:52.721755 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.721820 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:25:52.721755 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:25:52.721971 12237 server_base.cc:1034] running on GCE node
I20260929 03:25:52.722208 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:25:52.722497 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:25:52.723675 12237 hybrid_clock.cc:648] HybridClock initialized: now 1790652352723654 us; error 30 us; skew 500 ppm
Sep 29 03:25:52 dist-test-slave-fzwk krb5kdc[10798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:25:52.726444 12237 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:25:52.727013 12237 webserver.cc:460] Webserver started at http://127.27.68.67:45843/ using document root <none> and password file <none>
I20260929 03:25:52.727325 12237 fs_manager.cc:359] Metadata directory not provided
I20260929 03:25:52.727398 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:25:52.728849 12237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:25:52.729594 12252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.729772 12237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:25:52.729882 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "c67992b798e04feaa5ea20c375cbb5bd"
format_stamp: "Formatted at 2026-09-29 03:25:11 on dist-test-slave-fzwk"
server_key: "2370b7d8ead4c90bbae79dbe5e3b358f"
server_key_iv: "42d5521d904c33057703d6c6c8db7747"
server_key_version: "encryptionkey@0"
I20260929 03:25:52.730255 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:25:52.749218 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:25:52.751782 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:25:52.751961 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:25:52.752233 12237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:25:52.752565 12237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:25:52.752621 12237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.752665 12237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:25:52.752691 12237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:25:52.758754 12237 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:43495
I20260929 03:25:52.758833 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:43495 every 8 connection(s)
I20260929 03:25:52.759234 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:25:52.760407 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.759998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:60671)
0929 03:25:52.760175 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.760251 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
W20260929 03:25:52.760854 12366 heartbeater.cc:644] Failed to heartbeat to 127.27.68.126:35575 (0 consecutive failures): Network error: Failed to ping master at 127.27.68.126:35575: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
I20260929 03:25:52.761129 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.761041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:58819)
0929 03:25:52.761085 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.761111 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:25:52.761394 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:52.761338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:46651)
0929 03:25:52.761362 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:52.761381 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 03:25:52.761509 12366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:25:52.766639 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12237
I20260929 03:25:53.532928 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.532717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:48113)
0929 03:25:53.532830 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:53.532864 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260929 03:25:53.629776 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.629573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:56215)
0929 03:25:53.629690 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:53.629724 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260929 03:25:53.733945 12372 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:53.734090 12372 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:53.734505 12372 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
Sep 29 03:25:53 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.67@KRBTEST.COM
Sep 29 03:25:53 dist-test-slave-fzwk krb5kdc[10798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.66@KRBTEST.COM
I20260929 03:25:53.745653 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.735022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:47861 (local address 127.27.68.66:42119)
0929 03:25:53.735191 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:53.735200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:53.735269 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:25:53.735699 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:53.735701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:53.735739 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:53.735891 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:53.736646 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.737491 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.738573 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.739007 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.739159 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:53.742055 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:53.742060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:53.742060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:53.742073 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:53.744039 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744442 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:53.744444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:53.744444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:53.744576 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744827 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:53.744828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:53.744828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:53.745099 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:53.745181 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:53.745399 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 03:25:53.745533 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260929 03:25:53.745642 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.734790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43495 (local address 127.27.68.65:44479)
0929 03:25:53.734854 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:53.734869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:53.734950 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:53.735885 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:53.735888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:53.736304 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:53.736511 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:53.736517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.737703 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.737703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:53.738468 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:53.738471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.739193 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.739195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:53.739274 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:53.739947 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:53.739952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:53.741493 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:53.743866 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:53.743874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:53.743876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:53.744210 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:53.744492 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:53.744493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:53.744622 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:53.745061 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:53.745068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:53.745216 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:25:53.745334 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:25:53.745546 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":76}
I20260929 03:25:53.745679 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.734920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:44479 (local address 127.27.68.67:43495)
0929 03:25:53.735084 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:25:53.735098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:25:53.735126 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:25:53.735640 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:25:53.735642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:25:53.735698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:25:53.735814 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:25:53.736637 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.737608 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.738573 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.739087 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.739247 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:53.741604 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:25:53.741608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:25:53.741609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:25:53.741619 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:25:53.743776 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744294 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:53.744298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:53.744299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:53.744422 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744689 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:25:53.744690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:25:53.744690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:25:53.744991 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:25:53.745058 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:25:53.745406 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 03:25:53.745538 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":47,"threads_started":1}
I20260929 03:25:53.746181 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.734899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:42119 (local address 127.27.68.65:47861)
0929 03:25:53.735012 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:53.735023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:25:53.735090 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:25:53.735902 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:25:53.735903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:25:53.736302 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:25:53.736511 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:53.736516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.737585 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.737586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:53.738467 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:25:53.738469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:25:53.739125 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:25:53.739126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:25:53.739272 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:25:53.739877 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:25:53.739886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:25:53.741927 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:25:53.744136 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:53.744137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:53.744352 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:53.744651 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:25:53.744668 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:25:53.744669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:25:53.744758 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:25:53.745187 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:25:53.745188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:25:53.745224 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:25:53.745334 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:25:53.745636 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":156,"thread_start_us":50,"threads_started":1}
W20260929 03:25:53.746515 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
W20260929 03:25:53.746891 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
I20260929 03:25:53.746942 11989 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d; no voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63
I20260929 03:25:53.747085 12372 raft_consensus.cc:2726] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:53.762202 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:53.761963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:41905)
0929 03:25:53.762118 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:53.762163 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260929 03:25:54.533583 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:54.533363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:37737)
0929 03:25:54.533489 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:54.533536 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260929 03:25:54.630541 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:54.630225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:41197)
0929 03:25:54.630431 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:54.630494 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":60,"threads_started":1}
I20260929 03:25:54.762959 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:54.762745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:59249)
0929 03:25:54.762863 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:54.762902 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260929 03:25:55.534396 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:55.534131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:40333)
0929 03:25:55.534303 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:55.534340 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260929 03:25:55.631294 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:55.631025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:60525)
0929 03:25:55.631145 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:55.631187 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260929 03:25:55.666040 12382 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:55.666144 12382 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:55.666301 12382 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
W20260929 03:25:55.666815 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
W20260929 03:25:55.666916 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
I20260929 03:25:55.666997 11989 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d; no voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63
I20260929 03:25:55.667124 12382 raft_consensus.cc:2726] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:55.763705 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:55.763490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:58421)
0929 03:25:55.763614 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:55.763659 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260929 03:25:56.535082 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:56.534860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:47315)
0929 03:25:56.534986 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:56.535029 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260929 03:25:56.631976 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:56.631761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:50545)
0929 03:25:56.631894 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:56.631934 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260929 03:25:56.764447 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:56.764213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:33039)
0929 03:25:56.764359 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:56.764400 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260929 03:25:57.291405 12387 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:25:57.291514 12387 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:25:57.291647 12387 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
W20260929 03:25:57.292116 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
W20260929 03:25:57.292222 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
I20260929 03:25:57.292258 11989 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d; no voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63
I20260929 03:25:57.292362 12387 raft_consensus.cc:2726] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:25:57.535842 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:57.535632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:58065)
0929 03:25:57.535764 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:57.535794 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260929 03:25:57.632666 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:57.632404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:35583)
0929 03:25:57.632550 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:57.632586 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20260929 03:25:57.765288 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:57.765070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:53571)
0929 03:25:57.765199 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:57.765244 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260929 03:25:58.536501 12391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:58.536280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:38209)
0929 03:25:58.536419 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:58.536451 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260929 03:25:58.633384 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:58.633174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:54769)
0929 03:25:58.633297 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:58.633337 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260929 03:25:58.766125 12393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:58.765891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:44627)
0929 03:25:58.766041 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:58.766079 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260929 03:25:59.537199 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:59.536978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:34281)
0929 03:25:59.537109 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:59.537152 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260929 03:25:59.634091 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:59.633872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:40381)
0929 03:25:59.634001 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:59.634042 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260929 03:25:59.766958 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:25:59.766722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:55563)
0929 03:25:59.766863 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 03:25:59.766907 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260929 03:26:00.537947 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:00.537731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:33947)
0929 03:26:00.537852 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:00.537899 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260929 03:26:00.550673 12398 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:00.550824 12398 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:26:00.551030 12398 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
W20260929 03:26:00.551455 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
W20260929 03:26:00.551544 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
I20260929 03:26:00.551563 11989 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d; no voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63
I20260929 03:26:00.551633 12398 raft_consensus.cc:2726] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:26:00.634752 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:00.634524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:50583)
0929 03:26:00.634663 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:00.634707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260929 03:26:00.767817 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:00.767534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:41289)
0929 03:26:00.767726 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:00.767767 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":136,"threads_started":1}
I20260929 03:26:01.538616 12401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:01.538401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:56527)
0929 03:26:01.538533 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:01.538569 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260929 03:26:01.635465 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:01.635204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:36627)
0929 03:26:01.635374 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:01.635418 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260929 03:26:01.768450 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:01.768247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:48133)
0929 03:26:01.768367 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:01.768400 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260929 03:26:02.539407 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:02.539106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:46261)
0929 03:26:02.539297 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:02.539360 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260929 03:26:02.636159 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:02.635948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:38763)
0929 03:26:02.636069 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:02.636109 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260929 03:26:02.769346 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:02.769053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:34067)
0929 03:26:02.769217 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:02.769270 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260929 03:26:03.540099 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:03.539871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:49997)
0929 03:26:03.540006 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:03.540052 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260929 03:26:03.636896 12408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:03.636660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:54635)
0929 03:26:03.636803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:03.636847 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260929 03:26:03.770103 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:03.769850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:53719)
0929 03:26:03.770004 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:03.770044 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260929 03:26:04.540733 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:04.540519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:32857)
0929 03:26:04.540652 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:04.540688 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260929 03:26:04.637667 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:04.637363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:36371)
0929 03:26:04.637513 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:04.637601 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260929 03:26:04.771086 12412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:04.770833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:57085)
0929 03:26:04.770989 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:04.771036 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260929 03:26:05.541531 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:05.541288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:42045)
0929 03:26:05.541432 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:05.541484 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260929 03:26:05.638476 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:05.638242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:55977)
0929 03:26:05.638390 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:05.638428 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260929 03:26:05.771898 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:05.771647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:44425)
0929 03:26:05.771797 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:05.771847 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260929 03:26:06.542361 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:06.542129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:59349)
0929 03:26:06.542261 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:06.542306 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260929 03:26:06.639185 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:06.638916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:37287)
0929 03:26:06.639096 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:06.639137 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260929 03:26:06.772606 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:06.772369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:38873)
0929 03:26:06.772521 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:06.772558 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260929 03:26:07.167203 12419 raft_consensus.cc:482] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:07.167354 12419 raft_consensus.cc:504] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c67992b798e04feaa5ea20c375cbb5bd" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43495 } } peers { permanent_uuid: "e08a3b642ae5459eb15363c016113b63" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 42119 } } peers { permanent_uuid: "b203fee6638d4440bcc1d5f10623545d" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41735 } }
I20260929 03:26:07.167538 12419 leader_election.cc:288] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495), e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119)
W20260929 03:26:07.167951 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c67992b798e04feaa5ea20c375cbb5bd (127.27.68.67:43495): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
W20260929 03:26:07.168072 11989 leader_election.cc:341] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e08a3b642ae5459eb15363c016113b63 (127.27.68.66:42119): Not found: Tablet not found: c498780e935246c6b9ae03131d4150e5
I20260929 03:26:07.168107 11989 leader_election.cc:302] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b203fee6638d4440bcc1d5f10623545d; no voters: c67992b798e04feaa5ea20c375cbb5bd, e08a3b642ae5459eb15363c016113b63
I20260929 03:26:07.168200 12419 raft_consensus.cc:2726] T c498780e935246c6b9ae03131d4150e5 P b203fee6638d4440bcc1d5f10623545d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:26:07.543269 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:07.542978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.65:51389)
0929 03:26:07.543123 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:07.543170 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260929 03:26:07.640067 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:07.639789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.66:36575)
0929 03:26:07.639977 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:07.640021 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260929 03:26:07.767051 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 11939
I20260929 03:26:07.771972 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12104
I20260929 03:26:07.773669 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:07.773457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:35575 (local address 127.27.68.67:42841)
0929 03:26:07.773595 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:07.773628 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.126:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260929 03:26:07.776419 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12237
2026-09-29T03:26:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58286 ms)
[----------] 30 tests from SecurityITest (235524 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:07 dist-test-slave-fzwk krb5kdc[12425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:07 dist-test-slave-fzwk krb5kdc[12425](info): set up 2 sockets
Sep 29 03:26:07 dist-test-slave-fzwk krb5kdc[12425](info): commencing operation
krb5kdc: starting...
W20260929 03:26:08.815428 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:08 dist-test-slave-fzwk krb5kdc[12425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:08.932204 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46123
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:09.014837 12438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:09.015105 12438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:09.015146 12438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:09.016390 12438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:09.016467 12438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:09.016490 12438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:09.016510 12438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:09.016537 12438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:09.018100 12438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:09.018412 12438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:09.018651 12438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:09.023572 12444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.023568 12446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.023594 12438 server_base.cc:1034] running on GCE node
W20260929 03:26:09.023572 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.024001 12438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:09.024319 12438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:09.025501 12438 hybrid_clock.cc:648] HybridClock initialized: now 1790652369025486 us; error 44 us; skew 500 ppm
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:09.028123 12438 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:09.028631 12438 webserver.cc:460] Webserver started at http://127.27.68.126:38271/ using document root <none> and password file <none>
I20260929 03:26:09.028851 12438 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:09.028893 12438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:09.028986 12438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:09.030049 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "71274a928322494b8131e0a325a98dd5"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "4fa68e35f00b06464faf09a2a6807844"
server_key_iv: "8fe1a17a0a85e91afa17ce115b513825"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.030369 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "71274a928322494b8131e0a325a98dd5"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "4fa68e35f00b06464faf09a2a6807844"
server_key_iv: "8fe1a17a0a85e91afa17ce115b513825"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.031754 12438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:26:09.032516 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.032697 12438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:09.032790 12438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "71274a928322494b8131e0a325a98dd5"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "4fa68e35f00b06464faf09a2a6807844"
server_key_iv: "8fe1a17a0a85e91afa17ce115b513825"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.032852 12438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:09.044916 12438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:09.046746 12438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:09.046859 12438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:09.050784 12438 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46123
I20260929 03:26:09.050899 12505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46123 every 8 connection(s)
I20260929 03:26:09.051250 12438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:09.052208 12506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:09.054620 12506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: Bootstrap starting.
I20260929 03:26:09.055236 12506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:09.055539 12506 log.cc:824] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:09.056236 12506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: No bootstrap required, opened a new log
I20260929 03:26:09.057382 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12438
I20260929 03:26:09.057473 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:09.057663 27921 external_mini_cluster.cc:1428] Setting key 658ca41fda212c6c658523888caa526e
I20260929 03:26:09.058037 12506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } }
I20260929 03:26:09.058241 12506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:09.058287 12506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71274a928322494b8131e0a325a98dd5, State: Initialized, Role: FOLLOWER
I20260929 03:26:09.058480 12506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } }
I20260929 03:26:09.058585 12506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:09.058627 12506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:09.058686 12506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:09.059911 12506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } }
I20260929 03:26:09.060036 12506 leader_election.cc:302] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71274a928322494b8131e0a325a98dd5; no voters:
I20260929 03:26:09.060237 12506 leader_election.cc:288] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:09.060261 12511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:09.060387 12511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 71274a928322494b8131e0a325a98dd5, State: Running, Role: LEADER
I20260929 03:26:09.060521 12506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:09.060528 12511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } }
I20260929 03:26:09.061056 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71274a928322494b8131e0a325a98dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } } }
I20260929 03:26:09.061156 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:09.061401 12513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71274a928322494b8131e0a325a98dd5. Latest consensus state: current_term: 1 leader_uuid: "71274a928322494b8131e0a325a98dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71274a928322494b8131e0a325a98dd5" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46123 } } }
I20260929 03:26:09.061512 12513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:09.061691 12519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:09.062237 12519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:09.064019 12519 catalog_manager.cc:1347] Generated new cluster ID: 2877272e821e42658978da30f687f491
I20260929 03:26:09.064085 12519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:09.069707 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.058196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.27.68.126:46123)
0929 03:26:09.058437 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.058446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.058483 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:26:09.058587 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.058592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.058667 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.058811 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:09.059653 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.060830 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.062017 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.062619 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.062740 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.065852 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:09.065857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:09.065858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:09.065871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:09.067953 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.068407 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.068410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.068410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.068533 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.068769 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.068770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.068770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.069079 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:09.069236 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.069363 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.069500 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
I20260929 03:26:09.086887 12519 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:09.087505 12519 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:09.091899 12519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: Generated new TSK 0
I20260929 03:26:09.092109 12519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:26:09.123593 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:09.206530 12532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:09.206743 12532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:09.206765 12532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:09.207990 12532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:09.208035 12532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:09.208062 12532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:09.209501 12532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:09.209688 12532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:09.209941 12532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:09.214915 12540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.215006 12532 server_base.cc:1034] running on GCE node
W20260929 03:26:09.214911 12538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.215129 12537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.215380 12532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:09.215685 12532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:09.216857 12532 hybrid_clock.cc:648] HybridClock initialized: now 1790652369216845 us; error 46 us; skew 500 ppm
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:09.219461 12532 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:26:09.219987 12532 webserver.cc:460] Webserver started at http://127.27.68.65:40383/ using document root <none> and password file <none>
I20260929 03:26:09.220225 12532 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:09.220295 12532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:09.220443 12532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:09.221542 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "43299e28b12a497b81f9e585b1f76a74"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "102bf4fe0df3d85ad5181802f81317f6"
server_key_iv: "d009918abc53ac79199aa473f5e44dae"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.221918 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "43299e28b12a497b81f9e585b1f76a74"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "102bf4fe0df3d85ad5181802f81317f6"
server_key_iv: "d009918abc53ac79199aa473f5e44dae"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.223382 12532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:09.224151 12547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.224342 12532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:09.224447 12532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "43299e28b12a497b81f9e585b1f76a74"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "102bf4fe0df3d85ad5181802f81317f6"
server_key_iv: "d009918abc53ac79199aa473f5e44dae"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.224535 12532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:09.265929 12532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:09.269166 12532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:09.269373 12532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:09.269708 12532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:09.270135 12532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:09.270180 12532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.270205 12532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:09.270246 12532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.276389 12532 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:41889
I20260929 03:26:09.276427 12660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:41889 every 8 connection(s)
I20260929 03:26:09.276772 12532 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:26:09.279343 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12532
I20260929 03:26:09.279469 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:09.279680 27921 external_mini_cluster.cc:1428] Setting key 3a01ded427d9f270ff323228d2393ddc
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:09.287546 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.277601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52031 (local address 127.27.68.126:46123)
0929 03:26:09.277647 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.277649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.277865 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:26:09.278429 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.278430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.278447 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.278492 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:09.279570 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.280313 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.281325 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.281641 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.281711 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.284265 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:09.284267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:09.284267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:09.284275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:09.285714 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.286181 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.286183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.286183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.286271 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.286641 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.286642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.286643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.286843 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:09.286910 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.287349 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.287449 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:09.287765 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.277529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46123 (local address 127.27.68.65:52031)
0929 03:26:09.277731 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.277778 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.278305 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.278563 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.278567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.279167 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:09.279452 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.279477 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.280418 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.280419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.281228 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.281230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.281744 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.281745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.281917 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.282499 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:09.282509 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:09.284158 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:09.285821 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.285830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.285833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.286082 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.286383 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.286384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.286384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.286552 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.286925 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:09.286932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:09.287161 (+ 229us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.287316 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.287505 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
I20260929 03:26:09.288285 12661 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46123
I20260929 03:26:09.288404 12661 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:09.288601 12661 heartbeater.cc:511] Master 127.27.68.126:46123 requested a full tablet report, sending...
I20260929 03:26:09.289131 12470 ts_manager.cc:194] Registered new tserver with Master: 43299e28b12a497b81f9e585b1f76a74 (127.27.68.65:41889)
I20260929 03:26:09.289772 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:52031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:26:09.305804 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:09.389778 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:09.390017 12666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:09.390041 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:09.391381 12666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:09.391429 12666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:09.391458 12666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:09.393003 12666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:09.393268 12666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:09.393519 12666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:09.398640 12672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.398641 12671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.398689 12674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.398885 12666 server_base.cc:1034] running on GCE node
I20260929 03:26:09.399077 12666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:09.399361 12666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:09.400489 12666 hybrid_clock.cc:648] HybridClock initialized: now 1790652369400472 us; error 34 us; skew 500 ppm
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:09.403007 12666 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:26:09.403524 12666 webserver.cc:460] Webserver started at http://127.27.68.66:42913/ using document root <none> and password file <none>
I20260929 03:26:09.403754 12666 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:09.403800 12666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:09.403955 12666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:09.405045 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "f9449cf3df7a439bae36ab4232b673db"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "706d781a702a06224d41795936bcc926"
server_key_iv: "d9ac0d0e7661d7d06b648eee8b43d98b"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.405422 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "f9449cf3df7a439bae36ab4232b673db"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "706d781a702a06224d41795936bcc926"
server_key_iv: "d9ac0d0e7661d7d06b648eee8b43d98b"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.406772 12666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:09.407601 12681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.407799 12666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:26:09.407893 12666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "f9449cf3df7a439bae36ab4232b673db"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "706d781a702a06224d41795936bcc926"
server_key_iv: "d9ac0d0e7661d7d06b648eee8b43d98b"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.407971 12666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:09.423288 12666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:09.425273 12666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:09.425470 12666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:09.425767 12666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:09.426123 12666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:09.426160 12666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.426183 12666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:09.426235 12666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.432397 12666 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:39863
I20260929 03:26:09.432453 12794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:39863 every 8 connection(s)
I20260929 03:26:09.432821 12666 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:09.441130 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12666
I20260929 03:26:09.441226 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:09.441422 27921 external_mini_cluster.cc:1428] Setting key 5a4752305a002c08676b53731c96e30c
I20260929 03:26:09.442358 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.433636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:58689 (local address 127.27.68.126:46123)
0929 03:26:09.433697 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.433699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.433905 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:26:09.434391 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.434392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.434402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.434474 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:09.435496 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.436089 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.437066 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.437254 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.437305 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.439388 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:09.439389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:09.439390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:09.439397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:09.440870 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.441353 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.441354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.441355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.441422 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.441678 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.441680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.441680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.441805 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:09.441894 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.442179 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.442267 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:26:09.442592 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.433618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46123 (local address 127.27.68.66:58689)
0929 03:26:09.433768 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.433812 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.434305 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.434533 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.434537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.435154 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:09.435407 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.435414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.436162 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.436163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.436999 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.437001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.437322 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.437323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.437438 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.437907 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:09.437915 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:09.439300 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:09.440987 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.440993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.440994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.441257 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.441492 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.441493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.441494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.441608 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.441886 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:09.441892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:09.442042 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.442147 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.442315 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260929 03:26:09.443043 12795 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46123
I20260929 03:26:09.443159 12795 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:09.443390 12795 heartbeater.cc:511] Master 127.27.68.126:46123 requested a full tablet report, sending...
I20260929 03:26:09.443821 12470 ts_manager.cc:194] Registered new tserver with Master: f9449cf3df7a439bae36ab4232b673db (127.27.68.66:39863)
I20260929 03:26:09.444171 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:58689
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:26:09.466166 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:09.547909 12800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:09.548169 12800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:09.548213 12800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:09.549422 12800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:09.549499 12800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:09.549544 12800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:09.551064 12800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:46123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:09.551379 12800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:09.551625 12800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:09.556672 12806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.556708 12808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:09.556720 12805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:09.557003 12800 server_base.cc:1034] running on GCE node
I20260929 03:26:09.557194 12800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:09.557438 12800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:09.558579 12800 hybrid_clock.cc:648] HybridClock initialized: now 1790652369558564 us; error 29 us; skew 500 ppm
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:09.561403 12800 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:26:09.561910 12800 webserver.cc:460] Webserver started at http://127.27.68.67:45623/ using document root <none> and password file <none>
I20260929 03:26:09.562170 12800 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:09.562217 12800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:09.562322 12800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:09.563419 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "abb292eb6ecb442aa1db8d191995c72f"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "af3f036045bca1d1d3653d7df5346f64"
server_key_iv: "9a8d09e45e068ac78a4be1db0ebddeda"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.563803 12800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "abb292eb6ecb442aa1db8d191995c72f"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "af3f036045bca1d1d3653d7df5346f64"
server_key_iv: "9a8d09e45e068ac78a4be1db0ebddeda"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.565111 12800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:09.565847 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.566111 12800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:09.566223 12800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "abb292eb6ecb442aa1db8d191995c72f"
format_stamp: "Formatted at 2026-09-29 03:26:09 on dist-test-slave-fzwk"
server_key: "af3f036045bca1d1d3653d7df5346f64"
server_key_iv: "9a8d09e45e068ac78a4be1db0ebddeda"
server_key_version: "encryptionkey@0"
I20260929 03:26:09.566299 12800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:09.581313 12800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:09.583305 12800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:09.583465 12800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:09.583705 12800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:09.584043 12800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:09.584077 12800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.584129 12800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:09.584156 12800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:09.590142 12800 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:40255
I20260929 03:26:09.590186 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:40255 every 8 connection(s)
I20260929 03:26:09.590514 12800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:26:09.591219 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12800
I20260929 03:26:09.591357 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:09.591609 27921 external_mini_cluster.cc:1428] Setting key 8515294a6f968bfbf94f1757df1e454e
Sep 29 03:26:09 dist-test-slave-fzwk krb5kdc[12425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652369, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:09.600240 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.591558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:43563 (local address 127.27.68.126:46123)
0929 03:26:09.591645 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.591648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.591664 (+ 16us) server_negotiation.cc:407] Connection header received
0929 03:26:09.592286 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.592287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.592301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.592378 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:09.593542 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.594038 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.595030 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.595333 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.595399 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.597550 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:09.597552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:09.597552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:09.597559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:09.599049 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.599447 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.599449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.599449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.599504 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.599700 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.599700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.599701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.599799 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:09.599867 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.600102 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.600168 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:26:09.600402 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.591285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:46123 (local address 127.27.68.67:43563)
0929 03:26:09.591435 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.591496 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.592176 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.592572 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.592576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.593245 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:09.593443 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.593448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.594143 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.594144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.594935 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.594937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.595413 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.595413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.595540 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.596021 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:09.596029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:09.597458 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:09.599120 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.599125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.599126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.599395 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.599564 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:09.599565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:09.599566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:09.599660 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:09.599867 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:09.599871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:09.599982 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.600068 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.600186 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260929 03:26:09.600975 12929 heartbeater.cc:347] Connected to a master server at 127.27.68.126:46123
I20260929 03:26:09.601101 12929 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:09.601297 12929 heartbeater.cc:511] Master 127.27.68.126:46123 requested a full tablet report, sending...
I20260929 03:26:09.601733 12470 ts_manager.cc:194] Registered new tserver with Master: abb292eb6ecb442aa1db8d191995c72f (127.27.68.67:40255)
I20260929 03:26:09.602134 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:43563
I20260929 03:26:09.603785 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:09.611544 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.605208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.27.68.126:46123)
0929 03:26:09.605286 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.605288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.605400 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:26:09.605488 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.605489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.605499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.605544 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:09.606113 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.606587 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.607403 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.607593 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.607673 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.608886 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:09.608887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:09.608887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:09.608894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:09.610364 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.610693 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.610694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.610695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.610751 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:09.610998 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:09.610999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:09.611000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:09.611155 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:09.611319 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.611378 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.611440 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:26:09.613350 12470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:09.615021 12470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:09.621629 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.27.68.66:39863)
0929 03:26:09.617870 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.617882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.617891 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:09.617927 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.617929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.617941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.618030 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:09.618226 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619167 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.620457 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621195 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621360 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.621506 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.621544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":72,"threads_started":1}
I20260929 03:26:09.621620 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39863 (local address 127.0.0.1:48466)
0929 03:26:09.617624 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.617670 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.617744 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.618035 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.618038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.618040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:09.618130 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.618133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619305 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.619306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.620363 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.620366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621291 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.621372 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621387 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.621486 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.621543 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":74,"threads_started":1}
I20260929 03:26:09.621989 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:40255 (local address 127.0.0.1:47896)
0929 03:26:09.617654 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.617683 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.617762 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.618319 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.618319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.618322 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:09.618381 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.618383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619735 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.619736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.620766 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.620769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621732 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.621807 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621813 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.621862 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.621904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260929 03:26:09.622030 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.27.68.67:40255)
0929 03:26:09.618169 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.618173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.618184 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:09.618223 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.618225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.618240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.618322 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:09.618445 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619621 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.620851 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621570 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621644 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621696 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.621891 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.621938 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":88,"threads_started":1}
I20260929 03:26:09.622169 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41889 (local address 127.0.0.1:54818)
0929 03:26:09.618430 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.618440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.618505 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.618683 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.618684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.618686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:09.618724 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.618725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619911 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.619911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.620982 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.620985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621921 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.621977 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621981 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.622028 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.622083 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":67,"threads_started":1}
I20260929 03:26:09.622481 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.617654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.27.68.65:41889)
0929 03:26:09.617845 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.617855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.618512 (+ 657us) server_negotiation.cc:407] Connection header received
0929 03:26:09.618575 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.618577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.618592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.618673 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:09.618786 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.619810 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621075 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.621771 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.621835 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.621889 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.622350 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.622402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260929 03:26:09.622707 12863 tablet_service.cc:1467] Processing CreateTablet for tablet ac0bfb18d10d4807bb9216af1593be36 (DEFAULT_TABLE table=test-table [id=789941bd676d4100b83b46f0a787a994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:09.622665 12729 tablet_service.cc:1467] Processing CreateTablet for tablet ac0bfb18d10d4807bb9216af1593be36 (DEFAULT_TABLE table=test-table [id=789941bd676d4100b83b46f0a787a994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:09.623051 12863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0bfb18d10d4807bb9216af1593be36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:09.623051 12729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0bfb18d10d4807bb9216af1593be36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:09.623822 12595 tablet_service.cc:1467] Processing CreateTablet for tablet ac0bfb18d10d4807bb9216af1593be36 (DEFAULT_TABLE table=test-table [id=789941bd676d4100b83b46f0a787a994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:09.624128 12595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac0bfb18d10d4807bb9216af1593be36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:09.626511 12946 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Bootstrap starting.
I20260929 03:26:09.626928 12948 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Bootstrap starting.
I20260929 03:26:09.627096 12947 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Bootstrap starting.
I20260929 03:26:09.627338 12946 tablet_bootstrap.cc:654] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:09.627668 12948 tablet_bootstrap.cc:654] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:09.627786 12947 tablet_bootstrap.cc:654] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:09.627914 12946 log.cc:824] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:09.628273 12948 log.cc:824] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:09.628419 12947 log.cc:824] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:09.628898 12946 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: No bootstrap required, opened a new log
I20260929 03:26:09.629006 12948 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: No bootstrap required, opened a new log
I20260929 03:26:09.629006 12946 ts_tablet_manager.cc:1397] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:26:09.629071 12948 ts_tablet_manager.cc:1397] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:09.629248 12947 tablet_bootstrap.cc:492] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: No bootstrap required, opened a new log
I20260929 03:26:09.629377 12947 ts_tablet_manager.cc:1397] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:09.631022 12946 raft_consensus.cc:348] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631022 12948 raft_consensus.cc:348] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631249 12948 raft_consensus.cc:374] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:09.631256 12946 raft_consensus.cc:374] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:09.631295 12946 raft_consensus.cc:729] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb292eb6ecb442aa1db8d191995c72f, State: Initialized, Role: FOLLOWER
I20260929 03:26:09.631295 12948 raft_consensus.cc:729] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43299e28b12a497b81f9e585b1f76a74, State: Initialized, Role: FOLLOWER
I20260929 03:26:09.631186 12947 raft_consensus.cc:348] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631419 12946 consensus_queue.cc:260] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631419 12948 consensus_queue.cc:260] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631455 12947 raft_consensus.cc:374] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:09.631503 12947 raft_consensus.cc:729] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9449cf3df7a439bae36ab4232b673db, State: Initialized, Role: FOLLOWER
I20260929 03:26:09.631642 12947 consensus_queue.cc:260] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.631737 12948 ts_tablet_manager.cc:1428] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 03:26:09.631748 12946 ts_tablet_manager.cc:1428] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:26:09.631861 12929 heartbeater.cc:503] Master 127.27.68.126:46123 was elected leader, sending a full tablet report...
I20260929 03:26:09.631909 12661 heartbeater.cc:503] Master 127.27.68.126:46123 was elected leader, sending a full tablet report...
I20260929 03:26:09.631943 12947 ts_tablet_manager.cc:1428] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:26:09.631965 12795 heartbeater.cc:503] Master 127.27.68.126:46123 was elected leader, sending a full tablet report...
W20260929 03:26:09.683457 12796 tablet.cc:2369] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:26:09.777617 12662 tablet.cc:2369] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:26:09.841143 12930 tablet.cc:2369] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:26:09.847784 12953 raft_consensus.cc:482] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:09.847924 12953 raft_consensus.cc:504] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.848349 12953 leader_election.cc:288] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9449cf3df7a439bae36ab4232b673db (127.27.68.66:39863), abb292eb6ecb442aa1db8d191995c72f (127.27.68.67:40255)
I20260929 03:26:09.852682 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.848652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40495 (local address 127.27.68.67:40255)
0929 03:26:09.848708 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.848711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.848795 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:26:09.848841 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.848842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.848853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.848932 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:09.849166 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.849921 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.851663 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.852278 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.852380 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.852389 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.852561 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.852605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:26:09.852682 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.848652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52825 (local address 127.27.68.66:39863)
0929 03:26:09.848713 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.848715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.848795 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:26:09.848842 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.848842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.848853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.848932 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:09.849165 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.849928 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.851663 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.852278 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.852369 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.852380 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.852564 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.852605 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:26:09.852720 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.848553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:40255 (local address 127.27.68.65:40495)
0929 03:26:09.848702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.848715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.848786 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.848958 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.848959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.848962 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:09.849082 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.849084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.850028 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.850028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.851547 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.851551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.852384 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.852386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.852474 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.852480 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.852542 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.852603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":122,"thread_start_us":78,"threads_started":1}
I20260929 03:26:09.852722 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.848553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39863 (local address 127.27.68.65:52825)
0929 03:26:09.848703 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.848715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:09.848786 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:09.848958 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:09.848959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:09.848961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:09.849075 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.849078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.850027 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.850028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.851547 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:09.851551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.852386 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.852386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:09.852474 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.852482 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:26:09.852552 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:26:09.852603 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":110,"thread_start_us":82,"threads_started":1}
I20260929 03:26:09.853104 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0bfb18d10d4807bb9216af1593be36" candidate_uuid: "43299e28b12a497b81f9e585b1f76a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb292eb6ecb442aa1db8d191995c72f" is_pre_election: true
I20260929 03:26:09.853318 12883 raft_consensus.cc:2454] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43299e28b12a497b81f9e585b1f76a74 in term 0.
I20260929 03:26:09.853343 12749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0bfb18d10d4807bb9216af1593be36" candidate_uuid: "43299e28b12a497b81f9e585b1f76a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9449cf3df7a439bae36ab4232b673db" is_pre_election: true
I20260929 03:26:09.853470 12749 raft_consensus.cc:2454] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43299e28b12a497b81f9e585b1f76a74 in term 0.
I20260929 03:26:09.853600 12551 leader_election.cc:302] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43299e28b12a497b81f9e585b1f76a74, abb292eb6ecb442aa1db8d191995c72f; no voters:
I20260929 03:26:09.853742 12953 raft_consensus.cc:2781] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:09.853830 12953 raft_consensus.cc:482] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:09.853850 12953 raft_consensus.cc:3037] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:09.854629 12953 raft_consensus.cc:504] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.854784 12953 leader_election.cc:288] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [CANDIDATE]: Term 1 election: Requested vote from peers f9449cf3df7a439bae36ab4232b673db (127.27.68.66:39863), abb292eb6ecb442aa1db8d191995c72f (127.27.68.67:40255)
I20260929 03:26:09.854974 12883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0bfb18d10d4807bb9216af1593be36" candidate_uuid: "43299e28b12a497b81f9e585b1f76a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb292eb6ecb442aa1db8d191995c72f"
I20260929 03:26:09.854972 12749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0bfb18d10d4807bb9216af1593be36" candidate_uuid: "43299e28b12a497b81f9e585b1f76a74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9449cf3df7a439bae36ab4232b673db"
I20260929 03:26:09.855051 12749 raft_consensus.cc:3037] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:09.855051 12883 raft_consensus.cc:3037] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:09.855980 12883 raft_consensus.cc:2454] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43299e28b12a497b81f9e585b1f76a74 in term 1.
I20260929 03:26:09.855985 12749 raft_consensus.cc:2454] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43299e28b12a497b81f9e585b1f76a74 in term 1.
I20260929 03:26:09.856197 12549 leader_election.cc:302] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43299e28b12a497b81f9e585b1f76a74, f9449cf3df7a439bae36ab4232b673db; no voters:
I20260929 03:26:09.856292 12953 raft_consensus.cc:2781] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:09.856459 12953 raft_consensus.cc:686] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 1 LEADER]: Becoming Leader. State: Replica: 43299e28b12a497b81f9e585b1f76a74, State: Running, Role: LEADER
I20260929 03:26:09.856573 12953 consensus_queue.cc:237] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } }
I20260929 03:26:09.857434 12469 catalog_manager.cc:5495] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43299e28b12a497b81f9e585b1f76a74 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "43299e28b12a497b81f9e585b1f76a74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43299e28b12a497b81f9e585b1f76a74" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 41889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 } health_report { overall_health: UNKNOWN } } }
I20260929 03:26:09.904585 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.901959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.27.68.65:41889)
0929 03:26:09.902020 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:09.902023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:09.902034 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:09.902128 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:09.902129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:09.902148 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:09.902230 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:09.902385 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.902866 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.903700 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:09.903899 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:09.903983 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:09.904183 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:09.904294 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:09.904389 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:09.904445 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:26:09.904468 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:26:09.906476 12749 raft_consensus.cc:1261] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 1 FOLLOWER]: Refusing update from remote peer 43299e28b12a497b81f9e585b1f76a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:09.906507 12883 raft_consensus.cc:1261] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 1 FOLLOWER]: Refusing update from remote peer 43299e28b12a497b81f9e585b1f76a74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:09.906911 12957 consensus_queue.cc:1035] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9449cf3df7a439bae36ab4232b673db" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:09.907043 12953 consensus_queue.cc:1035] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb292eb6ecb442aa1db8d191995c72f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:09.909448 12960 mvcc.cc:205] Tried to move back new op lower bound from 7334512107134070784 to 7334512106933252096. Current Snapshot: MvccSnapshot[applied={T|T < 7334512107134070784}]
I20260929 03:26:09.909569 12962 mvcc.cc:205] Tried to move back new op lower bound from 7334512107134070784 to 7334512106933252096. Current Snapshot: MvccSnapshot[applied={T|T < 7334512107134070784}]
I20260929 03:26:09.909683 12961 mvcc.cc:205] Tried to move back new op lower bound from 7334512107134070784 to 7334512106933252096. Current Snapshot: MvccSnapshot[applied={T|T < 7334512107134070784}]
I20260929 03:26:09.911837 12469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:09.911945 12469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:09.912792 12469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71274a928322494b8131e0a325a98dd5: Sending DeleteTablet for 3 replicas of tablet ac0bfb18d10d4807bb9216af1593be36
I20260929 03:26:09.913161 12863 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0bfb18d10d4807bb9216af1593be36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:09 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:47896
I20260929 03:26:09.913179 12595 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0bfb18d10d4807bb9216af1593be36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:09 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:54818
I20260929 03:26:09.913410 12970 tablet_replica.cc:330] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74: stopping tablet replica
I20260929 03:26:09.913501 12729 tablet_service.cc:1514] Processing DeleteTablet for tablet ac0bfb18d10d4807bb9216af1593be36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:09 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:48466
I20260929 03:26:09.913563 12970 raft_consensus.cc:2229] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:26:09.913666 12970 raft_consensus.cc:2258] T ac0bfb18d10d4807bb9216af1593be36 P 43299e28b12a497b81f9e585b1f76a74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:09.913816 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12532
I20260929 03:26:09.914729 12971 tablet_replica.cc:330] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: stopping tablet replica
I20260929 03:26:09.914852 12971 raft_consensus.cc:2229] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:09.914969 12971 raft_consensus.cc:2258] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:09.915150 12972 tablet_replica.cc:330] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: stopping tablet replica
I20260929 03:26:09.915282 12972 raft_consensus.cc:2229] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:09.915400 12972 raft_consensus.cc:2258] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:09.915866 12972 ts_tablet_manager.cc:1905] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:09.916422 12971 ts_tablet_manager.cc:1905] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:09.917840 12972 ts_tablet_manager.cc:1918] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:09.917935 12972 log.cc:1196] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/ac0bfb18d10d4807bb9216af1593be36
I20260929 03:26:09.918198 12972 ts_tablet_manager.cc:1939] T ac0bfb18d10d4807bb9216af1593be36 P f9449cf3df7a439bae36ab4232b673db: Deleting consensus metadata
I20260929 03:26:09.918479 12971 ts_tablet_manager.cc:1918] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:09.918561 12971 log.cc:1196] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/ac0bfb18d10d4807bb9216af1593be36
I20260929 03:26:09.918673 12455 catalog_manager.cc:4841] TS f9449cf3df7a439bae36ab4232b673db (127.27.68.66:39863): tablet ac0bfb18d10d4807bb9216af1593be36 (table test-table [id=789941bd676d4100b83b46f0a787a994]) successfully deleted
I20260929 03:26:09.918825 12971 ts_tablet_manager.cc:1939] T ac0bfb18d10d4807bb9216af1593be36 P abb292eb6ecb442aa1db8d191995c72f: Deleting consensus metadata
I20260929 03:26:09.919332 12457 catalog_manager.cc:4841] TS abb292eb6ecb442aa1db8d191995c72f (127.27.68.67:40255): tablet ac0bfb18d10d4807bb9216af1593be36 (table test-table [id=789941bd676d4100b83b46f0a787a994]) successfully deleted
W20260929 03:26:09.920073 12455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:41889 (error 108)
I20260929 03:26:09.920667 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:09.920509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:41889 (local address 127.0.0.1:54830)
0929 03:26:09.920570 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:09.920626 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:26:09.920691 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12666
W20260929 03:26:09.920815 12455 catalog_manager.cc:4584] TS 43299e28b12a497b81f9e585b1f76a74 (127.27.68.65:41889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac0bfb18d10d4807bb9216af1593be36: Network error: Client connection negotiation failed: client connection to 127.27.68.65:41889: connect: Connection refused (error 111)
I20260929 03:26:09.925575 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12800
I20260929 03:26:09.930090 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12438
2026-09-29T03:26:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:09Z Disabled control of system clock
I20260929 03:26:09.958094 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:36461
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:36461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:10.041194 12977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:10.041441 12977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:10.041464 12977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:10.042789 12977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:10.042841 12977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:10.042855 12977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:10.042870 12977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:10.042922 12977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:10.044315 12977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:36461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:36461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:10.044586 12977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:10.044907 12977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:10.049871 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.049868 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.049868 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:10.050041 12977 server_base.cc:1034] running on GCE node
I20260929 03:26:10.050382 12977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:10.050719 12977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:10.051891 12977 hybrid_clock.cc:648] HybridClock initialized: now 1790652370051860 us; error 43 us; skew 500 ppm
I20260929 03:26:10.053196 12977 webserver.cc:460] Webserver started at http://127.27.68.126:38797/ using document root <none> and password file <none>
I20260929 03:26:10.053480 12977 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:10.053558 12977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:10.053710 12977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:10.054790 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "a0ec2b6289714ce9a0bc454e325b0559"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "e227c9cbc9e7064a809303a6eeffdc63"
server_key_iv: "cb90f025c99f0c650f07b438eb56839c"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.055176 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "a0ec2b6289714ce9a0bc454e325b0559"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "e227c9cbc9e7064a809303a6eeffdc63"
server_key_iv: "cb90f025c99f0c650f07b438eb56839c"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.056505 12977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:10.057257 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.057416 12977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:10.057511 12977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "a0ec2b6289714ce9a0bc454e325b0559"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "e227c9cbc9e7064a809303a6eeffdc63"
server_key_iv: "cb90f025c99f0c650f07b438eb56839c"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.057601 12977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:10.072364 12977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:10.072710 12977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:10.072873 12977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:10.076921 12977 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:36461
I20260929 03:26:10.076973 13043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:36461 every 8 connection(s)
I20260929 03:26:10.077353 12977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:10.078387 13044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:10.080830 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: Bootstrap starting.
I20260929 03:26:10.081457 13044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:10.081774 13044 log.cc:824] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:10.082597 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: No bootstrap required, opened a new log
I20260929 03:26:10.082726 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 12977
I20260929 03:26:10.082835 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:10.083019 27921 external_mini_cluster.cc:1428] Setting key c80de3e1e3cd2c60aab9298cc4d5f649
I20260929 03:26:10.085031 13044 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } }
I20260929 03:26:10.085228 13044 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:10.085269 13044 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ec2b6289714ce9a0bc454e325b0559, State: Initialized, Role: FOLLOWER
I20260929 03:26:10.085439 13044 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } }
I20260929 03:26:10.085556 13044 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:10.085592 13044 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:10.085646 13044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:10.086548 13044 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } }
I20260929 03:26:10.086678 13044 leader_election.cc:302] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ec2b6289714ce9a0bc454e325b0559; no voters:
I20260929 03:26:10.086869 13044 leader_election.cc:288] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:10.086936 13049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:10.087132 13049 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [term 1 LEADER]: Becoming Leader. State: Replica: a0ec2b6289714ce9a0bc454e325b0559, State: Running, Role: LEADER
I20260929 03:26:10.087136 13044 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:10.087350 13049 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } }
I20260929 03:26:10.089016 13050 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ec2b6289714ce9a0bc454e325b0559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } } }
I20260929 03:26:10.089126 13050 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:10.089223 13064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:10.089274 13064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:10.089403 13066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:10.089598 13066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:10.090212 13051 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ec2b6289714ce9a0bc454e325b0559. Latest consensus state: current_term: 1 leader_uuid: "a0ec2b6289714ce9a0bc454e325b0559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ec2b6289714ce9a0bc454e325b0559" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 36461 } } }
I20260929 03:26:10.090322 13051 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:10.091306 13066 catalog_manager.cc:1347] Generated new cluster ID: 38af10519e694fb3b1fefe2256cccf11
I20260929 03:26:10.091374 13066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:10.091480 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.083592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.27.68.126:36461)
0929 03:26:10.083849 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.083858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.083896 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:26:10.084976 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.084987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.085078 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.085249 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:10.085519 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.087379 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.088653 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.089314 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.089446 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.090610 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:10.090615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:10.090615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:10.090844 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:10.090888 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:10.090904 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:10.090997 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.091094 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.091180 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
I20260929 03:26:10.099661 13066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:10.100417 13066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:10.104727 13066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: Generated new TSK 0
I20260929 03:26:10.104938 13066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:10.108847 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--builtin_ntp_servers=127.27.68.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:10.192698 13068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:10.192921 13068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:10.192945 13068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:10.194222 13068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:10.194269 13068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:10.194298 13068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:10.195737 13068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:10.196010 13068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:10.196341 13068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:10.201280 13074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.201335 13076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:10.201464 13068 server_base.cc:1034] running on GCE node
W20260929 03:26:10.201272 13073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:10.201818 13068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:10.202112 13068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:10.203276 13068 hybrid_clock.cc:648] HybridClock initialized: now 1790652370203266 us; error 32 us; skew 500 ppm
I20260929 03:26:10.204766 13068 webserver.cc:460] Webserver started at http://127.27.68.65:35831/ using document root <none> and password file <none>
I20260929 03:26:10.205050 13068 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:10.205108 13068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:10.205237 13068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:10.206642 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "8b3054354fb7412cb20d33811c00429b"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "7a7e4c501197cb9da12ca301d8d275a7"
server_key_iv: "31a60912a8ebfad1f783f781c7da63e3"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.207057 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "8b3054354fb7412cb20d33811c00429b"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "7a7e4c501197cb9da12ca301d8d275a7"
server_key_iv: "31a60912a8ebfad1f783f781c7da63e3"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.208702 13068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:10.209678 13082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.209874 13068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:10.209965 13068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "8b3054354fb7412cb20d33811c00429b"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "7a7e4c501197cb9da12ca301d8d275a7"
server_key_iv: "31a60912a8ebfad1f783f781c7da63e3"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.210045 13068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:10.226478 13068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:10.226768 13068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:10.226874 13068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:10.227087 13068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:10.227510 13068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:10.227550 13068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.227610 13068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:10.227635 13068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.233446 13068 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:43731
I20260929 03:26:10.233525 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:43731 every 8 connection(s)
I20260929 03:26:10.233886 13068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:26:10.239162 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.234662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:40165 (local address 127.27.68.126:36461)
0929 03:26:10.234753 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.234756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.234971 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:26:10.235542 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.235544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.235556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.235608 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:10.236279 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.236814 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.237782 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.237971 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.238026 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.238851 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:10.238853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:10.238853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:10.238858 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:10.238886 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:10.238889 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:10.238946 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.239059 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.239089 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:26:10.239413 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.234624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36461 (local address 127.27.68.65:40165)
0929 03:26:10.234842 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.234877 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.235448 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.235672 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.235676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.235684 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:10.236202 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.236206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.236903 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.236904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.237699 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.237701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.238059 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.238059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.238176 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.238685 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:10.238686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:10.238715 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:10.238738 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:10.238957 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:10.238961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:10.238963 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.239049 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.239132 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260929 03:26:10.239974 13196 heartbeater.cc:347] Connected to a master server at 127.27.68.126:36461
I20260929 03:26:10.240098 13196 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:10.240285 13196 heartbeater.cc:511] Master 127.27.68.126:36461 requested a full tablet report, sending...
I20260929 03:26:10.240808 13008 ts_manager.cc:194] Registered new tserver with Master: 8b3054354fb7412cb20d33811c00429b (127.27.68.65:43731)
I20260929 03:26:10.241443 13008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.65:40165
I20260929 03:26:10.244103 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13068
I20260929 03:26:10.244207 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:10.244393 27921 external_mini_cluster.cc:1428] Setting key 5054667a3bbde1b78b06892bf2f85f8d
I20260929 03:26:10.245332 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--builtin_ntp_servers=127.27.68.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:10.328756 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:10.328972 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:10.328996 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:10.330219 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:10.330264 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:10.330291 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:10.331660 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:10.331872 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:10.332172 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:10.337282 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.337306 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.337296 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:10.337549 13199 server_base.cc:1034] running on GCE node
I20260929 03:26:10.337811 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:10.338052 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:10.339200 13199 hybrid_clock.cc:648] HybridClock initialized: now 1790652370339184 us; error 27 us; skew 500 ppm
I20260929 03:26:10.340629 13199 webserver.cc:460] Webserver started at http://127.27.68.66:45181/ using document root <none> and password file <none>
I20260929 03:26:10.340845 13199 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:10.340888 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:10.340982 13199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:10.342051 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "113bc9f3cbec482897fae8db9e8cf43e"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "50651d2e52ddaa06a3f43ac7df949cca"
server_key_iv: "4c23eeaa49fbd930433c3b6fd88b6545"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.342350 13199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "113bc9f3cbec482897fae8db9e8cf43e"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "50651d2e52ddaa06a3f43ac7df949cca"
server_key_iv: "4c23eeaa49fbd930433c3b6fd88b6545"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.343654 13199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:10.344364 13213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.344544 13199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:10.344636 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "113bc9f3cbec482897fae8db9e8cf43e"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "50651d2e52ddaa06a3f43ac7df949cca"
server_key_iv: "4c23eeaa49fbd930433c3b6fd88b6545"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.344696 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:10.371484 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:10.371795 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:10.371904 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:10.372128 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:10.372431 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:10.372467 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.372489 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:10.372502 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.378154 13199 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:45275
I20260929 03:26:10.378273 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:45275 every 8 connection(s)
I20260929 03:26:10.378600 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:26:10.380424 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13199
I20260929 03:26:10.380546 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:10.380720 27921 external_mini_cluster.cc:1428] Setting key 7a4f370478f7802c89de10edf5beb6e0
I20260929 03:26:10.381729 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--builtin_ntp_servers=127.27.68.84:44169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:26:10.384939 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.379435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:59485 (local address 127.27.68.126:36461)
0929 03:26:10.379506 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.379508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.379787 (+ 279us) server_negotiation.cc:407] Connection header received
0929 03:26:10.380366 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.380367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.380380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.380447 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:10.381323 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.381951 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.383006 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.383266 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.383416 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.384607 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:10.384608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:10.384609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:10.384614 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:10.384630 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:10.384635 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:10.384662 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.384786 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.384824 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:26:10.385478 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.379456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36461 (local address 127.27.68.66:59485)
0929 03:26:10.379662 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.379699 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.380252 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.380530 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.380534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.380568 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:10.381221 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.381227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.382061 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.382063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.382895 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.382897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.383457 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.383459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.383690 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.384415 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:10.384416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:10.384457 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:10.384487 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:10.384702 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:10.384707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:10.384710 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.385090 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.385207 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260929 03:26:10.386023 13327 heartbeater.cc:347] Connected to a master server at 127.27.68.126:36461
I20260929 03:26:10.386144 13327 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:10.386399 13327 heartbeater.cc:511] Master 127.27.68.126:36461 requested a full tablet report, sending...
I20260929 03:26:10.386811 13008 ts_manager.cc:194] Registered new tserver with Master: 113bc9f3cbec482897fae8db9e8cf43e (127.27.68.66:45275)
I20260929 03:26:10.387327 13008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.66:59485
W20260929 03:26:10.466903 13330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:10.467175 13330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:10.467212 13330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:10.468462 13330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:10.468524 13330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:10.468565 13330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:10.469978 13330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:10.470268 13330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:10.470607 13330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:26:10.475800 13330 server_base.cc:1034] running on GCE node
W20260929 03:26:10.475731 13336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.475731 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:10.475735 13338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:10.476146 13330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:10.476404 13330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:10.477578 13330 hybrid_clock.cc:648] HybridClock initialized: now 1790652370477558 us; error 31 us; skew 500 ppm
I20260929 03:26:10.478842 13330 webserver.cc:460] Webserver started at http://127.27.68.67:33581/ using document root <none> and password file <none>
I20260929 03:26:10.479108 13330 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:10.479175 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:10.479369 13330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:10.480458 13330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "4b7bd71997744abfa174b20bd7115ed2"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "0191017f63097e841f97276eb2b9ca75"
server_key_iv: "c7940d0df36bc6bb437b0f631d250a04"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.480822 13330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "4b7bd71997744abfa174b20bd7115ed2"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "0191017f63097e841f97276eb2b9ca75"
server_key_iv: "c7940d0df36bc6bb437b0f631d250a04"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.482163 13330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:10.482875 13344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.483052 13330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:10.483146 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "4b7bd71997744abfa174b20bd7115ed2"
format_stamp: "Formatted at 2026-09-29 03:26:10 on dist-test-slave-fzwk"
server_key: "0191017f63097e841f97276eb2b9ca75"
server_key_iv: "c7940d0df36bc6bb437b0f631d250a04"
server_key_version: "encryptionkey@0"
I20260929 03:26:10.483253 13330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:10.491302 13330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:10.491604 13330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:10.491752 13330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:10.491974 13330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:10.492321 13330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:10.492388 13330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.492429 13330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:10.492456 13330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:10.498289 13330 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:42069
I20260929 03:26:10.498355 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:42069 every 8 connection(s)
I20260929 03:26:10.498723 13330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:26:10.504469 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.499551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:35261 (local address 127.27.68.126:36461)
0929 03:26:10.499607 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.499609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.499853 (+ 244us) server_negotiation.cc:407] Connection header received
0929 03:26:10.500424 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.500425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.500440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.500509 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:10.501262 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.501789 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.502870 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.503081 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.503179 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.504141 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:10.504143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:10.504143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:10.504149 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:10.504170 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:10.504175 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:10.504241 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.504352 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.504386 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:10.504675 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.499506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:36461 (local address 127.27.68.67:35261)
0929 03:26:10.499688 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.499733 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.500298 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.500585 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.500590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.500599 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:10.501146 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.501151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.501898 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.501900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.502736 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.502738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.503191 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.503192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.503370 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.503951 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:10.503952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:10.503983 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:10.504007 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:10.504243 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:10.504249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:10.504253 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.504340 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.504448 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260929 03:26:10.505187 13458 heartbeater.cc:347] Connected to a master server at 127.27.68.126:36461
I20260929 03:26:10.505267 13458 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:10.505475 13458 heartbeater.cc:511] Master 127.27.68.126:36461 requested a full tablet report, sending...
I20260929 03:26:10.505888 13008 ts_manager.cc:194] Registered new tserver with Master: 4b7bd71997744abfa174b20bd7115ed2 (127.27.68.67:42069)
I20260929 03:26:10.506213 13008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.67:35261
I20260929 03:26:10.507681 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13330
I20260929 03:26:10.507781 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:10.507987 27921 external_mini_cluster.cc:1428] Setting key 2bbb2b55492354ae35bd0d449893e05f
I20260929 03:26:10.508836 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:10.512844 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.510183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.27.68.126:36461)
0929 03:26:10.510237 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.510239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.510364 (+ 125us) server_negotiation.cc:407] Connection header received
0929 03:26:10.510410 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.510410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.510420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.510466 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:10.510609 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.511039 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.511791 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.511968 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.512020 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.512669 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:10.512670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:10.512671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:10.512674 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:10.512683 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:10.512687 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:10.512724 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.512771 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.512790 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:26:10.514500 13008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:10.516357 13008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:10.524356 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.519088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.27.68.66:45275)
0929 03:26:10.519565 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.519568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.519577 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:10.519614 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.519616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.519632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.519908 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:10.519974 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.521080 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.523185 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.523931 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524013 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.524121 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.524229 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.524266 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":106,"threads_started":1}
I20260929 03:26:10.524374 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.518819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:45275 (local address 127.0.0.1:51430)
0929 03:26:10.519086 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.519099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.519178 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.519742 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.519745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.519747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:10.519818 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.519821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.521150 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.521151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.523096 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.523098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.524028 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.524119 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.524135 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.524213 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.524268 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260929 03:26:10.524705 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.518819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:43731 (local address 127.0.0.1:53260)
0929 03:26:10.518984 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.519037 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.519178 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.519745 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.519746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.519749 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:10.519850 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.519853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.521521 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.521522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.523434 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.523436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.524463 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.524528 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.524534 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.524602 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.524642 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260929 03:26:10.524792 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.519313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.27.68.65:43731)
0929 03:26:10.519515 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.519520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.519533 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:26:10.519587 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.519590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.519607 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.519692 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:10.519956 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.521304 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.523513 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.524047 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524128 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.524218 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.524642 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.524682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":124,"threads_started":1}
I20260929 03:26:10.525273 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.520293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:42069 (local address 127.0.0.1:43362)
0929 03:26:10.520941 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.520952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.521000 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.521580 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.521581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.521583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:10.521618 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.521619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.522829 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.522830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.523968 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.523972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.525035 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.525036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.525106 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.525109 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.525166 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.525206 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":86,"threads_started":1}
I20260929 03:26:10.525295 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.520377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.27.68.67:42069)
0929 03:26:10.521414 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.521418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.521425 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:26:10.521482 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.521484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.521500 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.521584 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:10.521684 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.522727 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524057 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.524852 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.524914 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.525002 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.525179 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.525217 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":74,"threads_started":1}
I20260929 03:26:10.525511 13130 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 (DEFAULT_TABLE table=test-table [id=e0d8abcbc8c241cf9e8bfc519fa19c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:10.525913 13130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadb2c9a8a4842508c6c06fe6dae5ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:10.527730 13261 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 (DEFAULT_TABLE table=test-table [id=e0d8abcbc8c241cf9e8bfc519fa19c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:10.528077 13261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadb2c9a8a4842508c6c06fe6dae5ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:10.529256 13472 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: Bootstrap starting.
I20260929 03:26:10.529867 13472 tablet_bootstrap.cc:654] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:10.530162 13472 log.cc:824] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:10.530867 13472 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: No bootstrap required, opened a new log
I20260929 03:26:10.530922 13472 ts_tablet_manager.cc:1397] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:10.531103 13474 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Bootstrap starting.
I20260929 03:26:10.531844 13474 tablet_bootstrap.cc:654] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:10.531785 13392 tablet_service.cc:1467] Processing CreateTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 (DEFAULT_TABLE table=test-table [id=e0d8abcbc8c241cf9e8bfc519fa19c2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:10.532109 13392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadb2c9a8a4842508c6c06fe6dae5ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:10.532218 13474 log.cc:824] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:10.532998 13474 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: No bootstrap required, opened a new log
I20260929 03:26:10.533082 13474 ts_tablet_manager.cc:1397] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:10.533684 13472 raft_consensus.cc:348] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.533918 13472 raft_consensus.cc:374] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:10.533968 13472 raft_consensus.cc:729] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3054354fb7412cb20d33811c00429b, State: Initialized, Role: FOLLOWER
I20260929 03:26:10.534127 13472 consensus_queue.cc:260] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.534384 13472 ts_tablet_manager.cc:1428] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260929 03:26:10.534511 13196 heartbeater.cc:503] Master 127.27.68.126:36461 was elected leader, sending a full tablet report...
I20260929 03:26:10.535207 13477 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Bootstrap starting.
I20260929 03:26:10.535959 13477 tablet_bootstrap.cc:654] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:10.536252 13477 log.cc:824] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:10.536227 13474 raft_consensus.cc:348] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.536401 13474 raft_consensus.cc:374] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:10.536444 13474 raft_consensus.cc:729] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113bc9f3cbec482897fae8db9e8cf43e, State: Initialized, Role: FOLLOWER
I20260929 03:26:10.536561 13474 consensus_queue.cc:260] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.536821 13474 ts_tablet_manager.cc:1428] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260929 03:26:10.536855 13327 heartbeater.cc:503] Master 127.27.68.126:36461 was elected leader, sending a full tablet report...
I20260929 03:26:10.536993 13477 tablet_bootstrap.cc:492] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: No bootstrap required, opened a new log
I20260929 03:26:10.537082 13477 ts_tablet_manager.cc:1397] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:10.539033 13477 raft_consensus.cc:348] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.539160 13477 raft_consensus.cc:374] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:10.539181 13477 raft_consensus.cc:729] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7bd71997744abfa174b20bd7115ed2, State: Initialized, Role: FOLLOWER
I20260929 03:26:10.539345 13477 consensus_queue.cc:260] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.539541 13477 ts_tablet_manager.cc:1428] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:26:10.539647 13458 heartbeater.cc:503] Master 127.27.68.126:36461 was elected leader, sending a full tablet report...
W20260929 03:26:10.629276 13328 tablet.cc:2369] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:26:10.631590 13476 raft_consensus.cc:482] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:10.631706 13476 raft_consensus.cc:504] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.632071 13476 leader_election.cc:288] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113bc9f3cbec482897fae8db9e8cf43e (127.27.68.66:45275), 4b7bd71997744abfa174b20bd7115ed2 (127.27.68.67:42069)
I20260929 03:26:10.636173 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.632368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:44925 (local address 127.27.68.66:45275)
0929 03:26:10.632417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.632419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.632621 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:26:10.632657 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.632658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.632666 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.632727 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:10.632868 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.633552 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635143 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.635765 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635848 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.635856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.636055 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.636099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:26:10.636173 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.632371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:52081 (local address 127.27.68.67:42069)
0929 03:26:10.632433 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.632435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.632445 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:10.632484 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.632484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.632491 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.632552 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:10.632745 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.633532 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635143 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.635765 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635848 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.635856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.636048 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.636096 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:26:10.636176 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.632248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:42069 (local address 127.27.68.65:52081)
0929 03:26:10.632323 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.632335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.632388 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.632574 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.632575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.632577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:10.632660 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.632662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.633636 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.633637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.635014 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.635019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.635867 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.635956 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.635963 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.636025 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.636079 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":53}
I20260929 03:26:10.636183 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.632288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:45275 (local address 127.27.68.65:44925)
0929 03:26:10.632496 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.632504 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:10.632581 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:10.632738 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:10.632739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:10.632740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:10.632800 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.632801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.633636 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.633637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.635014 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:10.635019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.635868 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.635869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:10.635956 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.635963 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:26:10.636025 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:26:10.636074 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":120,"thread_start_us":101,"threads_started":1}
I20260929 03:26:10.636586 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2c9a8a4842508c6c06fe6dae5ea7" candidate_uuid: "8b3054354fb7412cb20d33811c00429b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113bc9f3cbec482897fae8db9e8cf43e" is_pre_election: true
I20260929 03:26:10.636590 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2c9a8a4842508c6c06fe6dae5ea7" candidate_uuid: "8b3054354fb7412cb20d33811c00429b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7bd71997744abfa174b20bd7115ed2" is_pre_election: true
I20260929 03:26:10.636771 13281 raft_consensus.cc:2454] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3054354fb7412cb20d33811c00429b in term 0.
I20260929 03:26:10.636787 13412 raft_consensus.cc:2454] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b3054354fb7412cb20d33811c00429b in term 0.
I20260929 03:26:10.637030 13084 leader_election.cc:302] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7bd71997744abfa174b20bd7115ed2, 8b3054354fb7412cb20d33811c00429b; no voters:
I20260929 03:26:10.637177 13476 raft_consensus.cc:2781] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:10.637236 13476 raft_consensus.cc:482] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:10.637270 13476 raft_consensus.cc:3037] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:10.638005 13476 raft_consensus.cc:504] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.638167 13476 leader_election.cc:288] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [CANDIDATE]: Term 1 election: Requested vote from peers 113bc9f3cbec482897fae8db9e8cf43e (127.27.68.66:45275), 4b7bd71997744abfa174b20bd7115ed2 (127.27.68.67:42069)
I20260929 03:26:10.638322 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2c9a8a4842508c6c06fe6dae5ea7" candidate_uuid: "8b3054354fb7412cb20d33811c00429b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7bd71997744abfa174b20bd7115ed2"
I20260929 03:26:10.638363 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadb2c9a8a4842508c6c06fe6dae5ea7" candidate_uuid: "8b3054354fb7412cb20d33811c00429b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113bc9f3cbec482897fae8db9e8cf43e"
I20260929 03:26:10.638408 13412 raft_consensus.cc:3037] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:10.638451 13281 raft_consensus.cc:3037] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:10.639161 13412 raft_consensus.cc:2454] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3054354fb7412cb20d33811c00429b in term 1.
I20260929 03:26:10.639163 13281 raft_consensus.cc:2454] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b3054354fb7412cb20d33811c00429b in term 1.
I20260929 03:26:10.639382 13084 leader_election.cc:302] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7bd71997744abfa174b20bd7115ed2, 8b3054354fb7412cb20d33811c00429b; no voters:
I20260929 03:26:10.639503 13476 raft_consensus.cc:2781] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:10.639647 13476 raft_consensus.cc:686] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [term 1 LEADER]: Becoming Leader. State: Replica: 8b3054354fb7412cb20d33811c00429b, State: Running, Role: LEADER
I20260929 03:26:10.639744 13476 consensus_queue.cc:237] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } }
I20260929 03:26:10.640597 13008 catalog_manager.cc:5495] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b3054354fb7412cb20d33811c00429b (127.27.68.65). New cstate: current_term: 1 leader_uuid: "8b3054354fb7412cb20d33811c00429b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b3054354fb7412cb20d33811c00429b" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 43731 } health_report { overall_health: HEALTHY } } }
I20260929 03:26:10.645792 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.643401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.27.68.65:43731)
0929 03:26:10.643448 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:10.643450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:10.643460 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:10.643542 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:10.643543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:10.643573 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:10.643650 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:10.643752 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.644218 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.644991 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:10.645187 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:10.645262 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:10.645431 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:10.645527 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:10.645621 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:10.645662 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:26:10.645683 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:26:10.647581 13412 raft_consensus.cc:1261] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 1 FOLLOWER]: Refusing update from remote peer 8b3054354fb7412cb20d33811c00429b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:10.647585 13281 raft_consensus.cc:1261] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 1 FOLLOWER]: Refusing update from remote peer 8b3054354fb7412cb20d33811c00429b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:10.647948 13482 consensus_queue.cc:1035] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [LEADER]: Connected to new peer: Peer: permanent_uuid: "113bc9f3cbec482897fae8db9e8cf43e" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:10.648065 13476 consensus_queue.cc:1035] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7bd71997744abfa174b20bd7115ed2" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:10.650571 13485 mvcc.cc:205] Tried to move back new op lower bound from 7334512110169788416 to 7334512110141001728. Current Snapshot: MvccSnapshot[applied={T|T < 7334512110169788416}]
I20260929 03:26:10.650995 13487 mvcc.cc:205] Tried to move back new op lower bound from 7334512110169788416 to 7334512110141001728. Current Snapshot: MvccSnapshot[applied={T|T < 7334512110169788416}]
I20260929 03:26:10.653028 13008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:10.653141 13008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:10.653931 13008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0ec2b6289714ce9a0bc454e325b0559: Sending DeleteTablet for 3 replicas of tablet aadb2c9a8a4842508c6c06fe6dae5ea7
I20260929 03:26:10.654322 13261 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:10 UTC) from {username='slave'} at 127.0.0.1:51430
I20260929 03:26:10.654336 13392 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:10 UTC) from {username='slave'} at 127.0.0.1:43362
I20260929 03:26:10.654557 13495 tablet_replica.cc:330] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: stopping tablet replica
I20260929 03:26:10.654565 13130 tablet_service.cc:1514] Processing DeleteTablet for tablet aadb2c9a8a4842508c6c06fe6dae5ea7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:10 UTC) from {username='slave'} at 127.0.0.1:53260
I20260929 03:26:10.654667 13495 raft_consensus.cc:2229] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:10.654789 13495 raft_consensus.cc:2258] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:10.655088 13497 tablet_replica.cc:330] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 8b3054354fb7412cb20d33811c00429b: stopping tablet replica
I20260929 03:26:10.655176 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13068
I20260929 03:26:10.655812 13496 tablet_replica.cc:330] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: stopping tablet replica
I20260929 03:26:10.655922 13496 raft_consensus.cc:2229] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:10.656031 13496 raft_consensus.cc:2258] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:10.656723 13495 ts_tablet_manager.cc:1905] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:10.657603 13496 ts_tablet_manager.cc:1905] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:10.658344 13495 ts_tablet_manager.cc:1918] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:10.658437 13495 log.cc:1196] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/aadb2c9a8a4842508c6c06fe6dae5ea7
I20260929 03:26:10.658703 13495 ts_tablet_manager.cc:1939] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 113bc9f3cbec482897fae8db9e8cf43e: Deleting consensus metadata
I20260929 03:26:10.659150 12992 catalog_manager.cc:4841] TS 113bc9f3cbec482897fae8db9e8cf43e (127.27.68.66:45275): tablet aadb2c9a8a4842508c6c06fe6dae5ea7 (table test-table [id=e0d8abcbc8c241cf9e8bfc519fa19c2c]) successfully deleted
I20260929 03:26:10.659596 13496 ts_tablet_manager.cc:1918] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:10.659677 13496 log.cc:1196] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/aadb2c9a8a4842508c6c06fe6dae5ea7
I20260929 03:26:10.659907 13496 ts_tablet_manager.cc:1939] T aadb2c9a8a4842508c6c06fe6dae5ea7 P 4b7bd71997744abfa174b20bd7115ed2: Deleting consensus metadata
I20260929 03:26:10.660349 12993 catalog_manager.cc:4841] TS 4b7bd71997744abfa174b20bd7115ed2 (127.27.68.67:42069): tablet aadb2c9a8a4842508c6c06fe6dae5ea7 (table test-table [id=e0d8abcbc8c241cf9e8bfc519fa19c2c]) successfully deleted
W20260929 03:26:10.661130 12994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:43731 (error 108)
I20260929 03:26:10.661641 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13199
I20260929 03:26:10.661659 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:10.661514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:43731 (local address 127.0.0.1:53286)
0929 03:26:10.661572 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:10.661624 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:26:10.663340 12994 catalog_manager.cc:4584] TS 8b3054354fb7412cb20d33811c00429b (127.27.68.65:43731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aadb2c9a8a4842508c6c06fe6dae5ea7: Network error: Client connection negotiation failed: client connection to 127.27.68.65:43731: connect: Connection refused (error 111)
I20260929 03:26:10.666513 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13330
I20260929 03:26:10.670866 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 12977
2026-09-29T03:26:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (739 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:10 dist-test-slave-fzwk krb5kdc[13501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:10 dist-test-slave-fzwk krb5kdc[13501](info): set up 2 sockets
Sep 29 03:26:10 dist-test-slave-fzwk krb5kdc[13501](info): commencing operation
krb5kdc: starting...
W20260929 03:26:11.699206 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:11 dist-test-slave-fzwk krb5kdc[13501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:11.818512 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42091
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:11.900283 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:11.900498 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:11.900521 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:11.901650 13514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:11.901688 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:11.901702 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:11.901715 13514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:11.901727 13514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:11.903134 13514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:11.903355 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:11.903527 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:11.908368 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:11.908453 13514 server_base.cc:1034] running on GCE node
W20260929 03:26:11.908368 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:11.908476 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:11.908907 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:11.909258 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:11.910413 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790652371910389 us; error 36 us; skew 500 ppm
Sep 29 03:26:11 dist-test-slave-fzwk krb5kdc[13501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652371, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:11.913143 13514 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:11.913597 13514 webserver.cc:460] Webserver started at http://127.27.68.126:45797/ using document root <none> and password file <none>
I20260929 03:26:11.913836 13514 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:11.913905 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:11.914052 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:11.915148 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "6fafef498e5d4ab9a14cd8b93a4e0474"
format_stamp: "Formatted at 2026-09-29 03:26:11 on dist-test-slave-fzwk"
server_key: "e00961d37628652079653d44d42eb621"
server_key_iv: "5db3d038a37c34fbd59d1ab01ed1cb20"
server_key_version: "encryptionkey@0"
I20260929 03:26:11.915589 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "6fafef498e5d4ab9a14cd8b93a4e0474"
format_stamp: "Formatted at 2026-09-29 03:26:11 on dist-test-slave-fzwk"
server_key: "e00961d37628652079653d44d42eb621"
server_key_iv: "5db3d038a37c34fbd59d1ab01ed1cb20"
server_key_version: "encryptionkey@0"
I20260929 03:26:11.916935 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:11.917639 13529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:11.917824 13514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:11.917932 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "6fafef498e5d4ab9a14cd8b93a4e0474"
format_stamp: "Formatted at 2026-09-29 03:26:11 on dist-test-slave-fzwk"
server_key: "e00961d37628652079653d44d42eb621"
server_key_iv: "5db3d038a37c34fbd59d1ab01ed1cb20"
server_key_version: "encryptionkey@0"
I20260929 03:26:11.918018 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:11.931630 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:11.933576 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:11.933738 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:11.937757 13514 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42091
I20260929 03:26:11.937810 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42091 every 8 connection(s)
I20260929 03:26:11.938196 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:11.939114 13582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:11.941443 13582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: Bootstrap starting.
I20260929 03:26:11.942030 13582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:11.942338 13582 log.cc:824] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:11.943033 13582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: No bootstrap required, opened a new log
I20260929 03:26:11.943531 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13514
I20260929 03:26:11.943641 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:11.943832 27921 external_mini_cluster.cc:1428] Setting key ca234bf95c024f0a534f176efe049c0b
I20260929 03:26:11.945580 13582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } }
I20260929 03:26:11.945768 13582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:11.945813 13582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fafef498e5d4ab9a14cd8b93a4e0474, State: Initialized, Role: FOLLOWER
I20260929 03:26:11.945955 13582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } }
I20260929 03:26:11.946055 13582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:11.946090 13582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:11.946143 13582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:11.946811 13582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } }
I20260929 03:26:11.946899 13582 leader_election.cc:302] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fafef498e5d4ab9a14cd8b93a4e0474; no voters:
I20260929 03:26:11.947125 13582 leader_election.cc:288] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:11.947155 13587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:11.947295 13587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [term 1 LEADER]: Becoming Leader. State: Replica: 6fafef498e5d4ab9a14cd8b93a4e0474, State: Running, Role: LEADER
I20260929 03:26:11.947381 13587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } }
I20260929 03:26:11.947415 13582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:11.947731 13587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fafef498e5d4ab9a14cd8b93a4e0474. Latest consensus state: current_term: 1 leader_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } } }
I20260929 03:26:11.947741 13589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fafef498e5d4ab9a14cd8b93a4e0474" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42091 } } }
I20260929 03:26:11.947816 13587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:11.947845 13589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474 [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:11.949209 13603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:11.949270 13603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:11.949323 13593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:11.949476 13593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:11.951004 13593 catalog_manager.cc:1347] Generated new cluster ID: 6287e39b1f954962b2ace3bbd5005b05
I20260929 03:26:11.951076 13593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:11 dist-test-slave-fzwk krb5kdc[13501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:11.955561 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:11.944360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.27.68.126:42091)
0929 03:26:11.944533 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:11.944541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:11.944567 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:26:11.945511 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:11.945518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:11.945570 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:11.945686 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:11.946506 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:11.947693 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:11.948825 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:11.949436 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:11.949551 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:11.952222 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:11.952227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:11.952227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:11.952240 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:11.953979 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:11.954346 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:11.954349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:11.954349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:11.954456 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:11.954633 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:11.954634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:11.954635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:11.954935 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:11.955057 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:11.955181 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 03:26:11.955340 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260929 03:26:11.960907 13593 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:11.961711 13593 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:11.973807 13593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: Generated new TSK 0
I20260929 03:26:11.974051 13593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:26:12.006796 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:12.089161 13608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:12.089375 13608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:12.089398 13608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:12.090503 13608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:12.090544 13608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:12.090572 13608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:12.092103 13608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:12.092312 13608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:12.092502 13608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:12.097471 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.097471 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.097468 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:12.097697 13608 server_base.cc:1034] running on GCE node
I20260929 03:26:12.097904 13608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:12.098160 13608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:12.099273 13608 hybrid_clock.cc:648] HybridClock initialized: now 1790652372099263 us; error 41 us; skew 500 ppm
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:12.101784 13608 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:26:12.102221 13608 webserver.cc:460] Webserver started at http://127.27.68.65:45469/ using document root <none> and password file <none>
I20260929 03:26:12.102452 13608 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:12.102499 13608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:12.102641 13608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:12.103757 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "03a2d9432bba4aebaa8261f106dd14b9"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "e6d2de11d637e6c1e8c182c8a4adf7ca"
server_key_iv: "2eb304729b76542bb2d670fd54004045"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.104122 13608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "03a2d9432bba4aebaa8261f106dd14b9"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "e6d2de11d637e6c1e8c182c8a4adf7ca"
server_key_iv: "2eb304729b76542bb2d670fd54004045"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.105536 13608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:12.106256 13623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.106432 13608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:12.106544 13608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "03a2d9432bba4aebaa8261f106dd14b9"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "e6d2de11d637e6c1e8c182c8a4adf7ca"
server_key_iv: "2eb304729b76542bb2d670fd54004045"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.106621 13608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:12.116166 13608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:12.118496 13608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:12.118661 13608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:12.118896 13608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:12.119252 13608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:12.119319 13608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.119361 13608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:12.119390 13608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.125383 13608 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:36055
I20260929 03:26:12.125424 13736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:36055 every 8 connection(s)
I20260929 03:26:12.125752 13608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:12.131875 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13608
I20260929 03:26:12.131985 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:12.132208 27921 external_mini_cluster.cc:1428] Setting key ccf8f43bfc1dccebc2eba8e28e87dde0
I20260929 03:26:12.135062 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.126495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:54843 (local address 127.27.68.126:42091)
0929 03:26:12.126541 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.126544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.126749 (+ 205us) server_negotiation.cc:407] Connection header received
0929 03:26:12.127252 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.127254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.127269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.127324 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:12.128250 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.128765 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.129706 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.129889 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.129939 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.132041 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:12.132043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:12.132043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:12.132053 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:12.133610 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.134046 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.134047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.134048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.134112 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.134375 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.134376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.134376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.134504 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:12.134580 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.134866 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.134962 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:12.135304 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.126473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42091 (local address 127.27.68.65:54843)
0929 03:26:12.126623 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.126660 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.127135 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.127387 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.127390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.127964 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:12.128170 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.128175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.128847 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.128847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.129633 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.129635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.129965 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.129966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.130088 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.130555 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:12.130564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:12.131902 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:12.133699 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.133707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.133709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.133951 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.134183 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.134184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.134185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.134305 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.134580 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:12.134589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:12.134733 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.134846 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.135026 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260929 03:26:12.135855 13737 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42091
I20260929 03:26:12.135974 13737 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:12.136160 13737 heartbeater.cc:511] Master 127.27.68.126:42091 requested a full tablet report, sending...
I20260929 03:26:12.136678 13546 ts_manager.cc:194] Registered new tserver with Master: 03a2d9432bba4aebaa8261f106dd14b9 (127.27.68.65:36055)
I20260929 03:26:12.137303 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:54843
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:26:12.158015 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:12.239586 13742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:12.239800 13742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:12.239822 13742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:12.240931 13742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:12.240967 13742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:12.240993 13742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:12.242515 13742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:12.242719 13742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:12.242904 13742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:12.247812 13750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.247812 13748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:12.247987 13742 server_base.cc:1034] running on GCE node
W20260929 03:26:12.247807 13747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:12.248284 13742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:12.248641 13742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:12.249804 13742 hybrid_clock.cc:648] HybridClock initialized: now 1790652372249783 us; error 39 us; skew 500 ppm
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:12.252382 13742 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:26:12.252861 13742 webserver.cc:460] Webserver started at http://127.27.68.66:41917/ using document root <none> and password file <none>
I20260929 03:26:12.253113 13742 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:12.253159 13742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:12.253326 13742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:12.254397 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "6ea5e17efec5416fa0c809c01e91ef15"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "72ff62abab69ed20473b4b6a5c1d59a9"
server_key_iv: "08815e094b94f170c833606ff48c0d3e"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.254766 13742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "6ea5e17efec5416fa0c809c01e91ef15"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "72ff62abab69ed20473b4b6a5c1d59a9"
server_key_iv: "08815e094b94f170c833606ff48c0d3e"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.256287 13742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:12.257035 13757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.257213 13742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.257320 13742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "6ea5e17efec5416fa0c809c01e91ef15"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "72ff62abab69ed20473b4b6a5c1d59a9"
server_key_iv: "08815e094b94f170c833606ff48c0d3e"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.257397 13742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:12.268750 13742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:12.270679 13742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:12.270848 13742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:12.271100 13742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:12.271477 13742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:12.271533 13742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.271579 13742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:12.271605 13742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.277565 13742 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:44709
I20260929 03:26:12.277608 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:44709 every 8 connection(s)
I20260929 03:26:12.277971 13742 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:26:12.283022 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13742
I20260929 03:26:12.283154 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:12.283376 27921 external_mini_cluster.cc:1428] Setting key 58d548818143c70a6d11614076377383
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:12.288965 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.278786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:48297 (local address 127.27.68.126:42091)
0929 03:26:12.278843 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.278845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.279117 (+ 272us) server_negotiation.cc:407] Connection header received
0929 03:26:12.279952 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.279953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.279968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.280043 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:12.281127 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.281943 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.282979 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.283272 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.283364 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.285800 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:12.285801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:12.285802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:12.285809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:12.287557 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.287933 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.287935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.287935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.288003 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.288242 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.288243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.288243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.288404 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:12.288477 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.288727 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.288820 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:12.288976 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.278795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42091 (local address 127.27.68.66:48297)
0929 03:26:12.278968 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.279025 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.279784 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.280121 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.280126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.280818 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:12.281026 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.281030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.282059 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.282060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.282875 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.282877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.283377 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.283378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.283520 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.284148 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:12.284157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:12.285692 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:12.287637 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.287644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.287646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.287872 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.288077 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.288077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.288078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.288175 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.288471 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:12.288475 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:12.288594 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.288677 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.288804 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260929 03:26:12.289579 13871 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42091
I20260929 03:26:12.289697 13871 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:12.289877 13871 heartbeater.cc:511] Master 127.27.68.126:42091 requested a full tablet report, sending...
I20260929 03:26:12.290304 13546 ts_manager.cc:194] Registered new tserver with Master: 6ea5e17efec5416fa0c809c01e91ef15 (127.27.68.66:44709)
I20260929 03:26:12.290707 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:48297
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:26:12.309901 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:12.392057 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:12.392298 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:12.392324 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:12.393528 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:12.393579 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:12.393605 13876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:12.395112 13876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:12.395388 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:12.395589 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:12.400609 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.400609 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.400633 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:12.400708 13876 server_base.cc:1034] running on GCE node
I20260929 03:26:12.401021 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:12.401294 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:12.402442 13876 hybrid_clock.cc:648] HybridClock initialized: now 1790652372402421 us; error 40 us; skew 500 ppm
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:12.405011 13876 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:26:12.405536 13876 webserver.cc:460] Webserver started at http://127.27.68.67:42857/ using document root <none> and password file <none>
I20260929 03:26:12.405779 13876 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:12.405826 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:12.405997 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:12.407094 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "24578260d2fa4daaa7884a6392f35696"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "6a020a7d7976b3932c8238d042ad7b4e"
server_key_iv: "d4cb34604e069e89bf95095bc1a95082"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.407536 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "24578260d2fa4daaa7884a6392f35696"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "6a020a7d7976b3932c8238d042ad7b4e"
server_key_iv: "d4cb34604e069e89bf95095bc1a95082"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.408901 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:12.409724 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.410018 13876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:12.410101 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "24578260d2fa4daaa7884a6392f35696"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "6a020a7d7976b3932c8238d042ad7b4e"
server_key_iv: "d4cb34604e069e89bf95095bc1a95082"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.410197 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:12.418660 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:12.420702 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:12.420862 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:12.421096 13876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:12.421451 13876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:12.421514 13876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.421554 13876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:12.421581 13876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.427531 13876 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:42341
I20260929 03:26:12.427585 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:42341 every 8 connection(s)
I20260929 03:26:12.427924 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:26:12 dist-test-slave-fzwk krb5kdc[13501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:12.434612 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 13876
I20260929 03:26:12.434747 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:12.434978 27921 external_mini_cluster.cc:1428] Setting key 40282057535c99b906a812fa68875164
I20260929 03:26:12.437546 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.428655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:43031 (local address 127.27.68.126:42091)
0929 03:26:12.428730 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.428732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.428954 (+ 222us) server_negotiation.cc:407] Connection header received
0929 03:26:12.429489 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.429491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.429504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.429600 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:12.430586 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.431108 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.432187 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.432409 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.432497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.434601 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:12.434603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:12.434603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:12.434614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:12.436395 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.436788 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.436789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.436789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.436842 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.437029 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.437029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.437030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.437142 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:12.437192 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.437421 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.437481 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:26:12.437776 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.428654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:42091 (local address 127.27.68.67:43031)
0929 03:26:12.428825 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.428862 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.429378 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.429685 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.429689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.430265 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:12.430471 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.430475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.431246 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.431247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.432084 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.432086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.432510 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.432512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.432647 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.433123 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:12.433131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:12.434487 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:12.436467 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.436473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.436475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.436720 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.436882 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:12.436882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:12.436882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:12.436979 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:12.437200 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:12.437204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:12.437326 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.437411 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.437548 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260929 03:26:12.438230 14005 heartbeater.cc:347] Connected to a master server at 127.27.68.126:42091
I20260929 03:26:12.438328 14005 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:12.438480 14005 heartbeater.cc:511] Master 127.27.68.126:42091 requested a full tablet report, sending...
I20260929 03:26:12.438843 13546 ts_manager.cc:194] Registered new tserver with Master: 24578260d2fa4daaa7884a6392f35696 (127.27.68.67:42341)
I20260929 03:26:12.439286 13546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:43031
I20260929 03:26:12.447144 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:12.455078 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.448836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.27.68.126:42091)
0929 03:26:12.448876 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.448878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.449028 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:26:12.449119 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.449119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.449130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.449204 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:12.449818 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.450321 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.451089 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.451356 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.451444 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.452802 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:12.452804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:12.452804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:12.452813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:12.453959 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.454373 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.454374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.454375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.454431 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:12.454626 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:12.454626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:12.454627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:12.454759 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:12.454843 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.454946 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.455001 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:26:12.456792 13546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:12.458362 13546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:12.464931 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.460675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:44709 (local address 127.0.0.1:50028)
0929 03:26:12.460857 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.460918 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.460990 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.461323 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.461327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.461329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:12.461445 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.461447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.462442 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.462443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.463621 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.463625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.464573 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.464676 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.464692 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.464779 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.464843 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260929 03:26:12.464931 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.460771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.27.68.66:44709)
0929 03:26:12.460957 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.460963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.460976 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:26:12.461131 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.461133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.461151 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.461223 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:12.461535 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.462336 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.463718 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.464481 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464567 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.464631 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.464802 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.464844 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260929 03:26:12.465167 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.460699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:42341 (local address 127.0.0.1:36662)
0929 03:26:12.461094 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.461104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.461163 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.461527 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.461528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.461530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:12.461582 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.461583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.462936 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.462938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.464037 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.464040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.464919 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.464984 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.464988 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.465054 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.465100 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":48,"threads_started":1}
I20260929 03:26:12.465482 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.460969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.27.68.67:42341)
0929 03:26:12.461285 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.461289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.461299 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:12.461339 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.461342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.461358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.461443 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:12.461660 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.462814 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464135 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.464771 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464833 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.464886 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.465359 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.465406 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":88,"threads_started":1}
I20260929 03:26:12.465857 13805 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca (DEFAULT_TABLE table=test-table [id=4c528050b99b49239fee4412d08a3478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:12.466086 13939 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca (DEFAULT_TABLE table=test-table [id=4c528050b99b49239fee4412d08a3478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:12.466230 13805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbc23c3fad4c598e2b61d812a6beca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:12.466368 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbc23c3fad4c598e2b61d812a6beca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:12.466588 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.460894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:36055 (local address 127.0.0.1:57678)
0929 03:26:12.461138 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.461146 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.461192 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.461603 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.461604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.461606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:12.461648 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.461649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.464145 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.464146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.465354 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.465357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.466336 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.466337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.466406 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.466410 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.466476 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.466522 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":52,"threads_started":1}
I20260929 03:26:12.466639 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.461099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.27.68.65:36055)
0929 03:26:12.461367 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.461372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.461382 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:12.461419 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.461421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.461436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.461512 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:12.461716 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.462829 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.465492 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.466234 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.466314 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.466372 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.466501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.466542 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
I20260929 03:26:12.469192 14019 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Bootstrap starting.
I20260929 03:26:12.469224 13671 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca (DEFAULT_TABLE table=test-table [id=4c528050b99b49239fee4412d08a3478]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:12.469609 13671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfbc23c3fad4c598e2b61d812a6beca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:12.469882 14019 tablet_bootstrap.cc:654] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:12.470359 14019 log.cc:824] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:12.470867 14021 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Bootstrap starting.
I20260929 03:26:12.471277 14019 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: No bootstrap required, opened a new log
I20260929 03:26:12.471354 14019 ts_tablet_manager.cc:1397] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:12.471633 14021 tablet_bootstrap.cc:654] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:12.472185 14021 log.cc:824] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:12.472900 14023 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Bootstrap starting.
I20260929 03:26:12.472983 14021 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: No bootstrap required, opened a new log
I20260929 03:26:12.473059 14021 ts_tablet_manager.cc:1397] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:26:12.473639 14023 tablet_bootstrap.cc:654] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:12.473533 14019 raft_consensus.cc:348] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.473717 14019 raft_consensus.cc:374] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:12.473758 14019 raft_consensus.cc:729] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ea5e17efec5416fa0c809c01e91ef15, State: Initialized, Role: FOLLOWER
I20260929 03:26:12.473874 14019 consensus_queue.cc:260] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.474165 14019 ts_tablet_manager.cc:1428] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:26:12.474201 14023 log.cc:824] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:12.474464 13871 heartbeater.cc:503] Master 127.27.68.126:42091 was elected leader, sending a full tablet report...
I20260929 03:26:12.475026 14023 tablet_bootstrap.cc:492] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: No bootstrap required, opened a new log
I20260929 03:26:12.475117 14023 ts_tablet_manager.cc:1397] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:12.475265 14021 raft_consensus.cc:348] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.475445 14021 raft_consensus.cc:374] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:12.475479 14021 raft_consensus.cc:729] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24578260d2fa4daaa7884a6392f35696, State: Initialized, Role: FOLLOWER
I20260929 03:26:12.475626 14021 consensus_queue.cc:260] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.475901 14021 ts_tablet_manager.cc:1428] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:26:12.475929 14005 heartbeater.cc:503] Master 127.27.68.126:42091 was elected leader, sending a full tablet report...
I20260929 03:26:12.477203 14023 raft_consensus.cc:348] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.477336 14023 raft_consensus.cc:374] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:12.477357 14023 raft_consensus.cc:729] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a2d9432bba4aebaa8261f106dd14b9, State: Initialized, Role: FOLLOWER
I20260929 03:26:12.477489 14023 consensus_queue.cc:260] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.477764 14023 ts_tablet_manager.cc:1428] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:26:12.477764 13737 heartbeater.cc:503] Master 127.27.68.126:42091 was elected leader, sending a full tablet report...
W20260929 03:26:12.528612 13872 tablet.cc:2369] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:26:12.626616 13738 tablet.cc:2369] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:26:12.678651 14006 tablet.cc:2369] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:26:12.697587 14027 raft_consensus.cc:482] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:12.697712 14027 raft_consensus.cc:504] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.698041 14027 leader_election.cc:288] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ea5e17efec5416fa0c809c01e91ef15 (127.27.68.66:44709), 24578260d2fa4daaa7884a6392f35696 (127.27.68.67:42341)
I20260929 03:26:12.702142 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.698329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:56273 (local address 127.27.68.66:44709)
0929 03:26:12.698379 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.698381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.698578 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:26:12.698657 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.698658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.698668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.698722 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:12.698902 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.699649 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701178 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.701771 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.701862 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.702027 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.702074 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:26:12.702142 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.698345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:45681 (local address 127.27.68.67:42341)
0929 03:26:12.698386 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.698387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.698517 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:26:12.698595 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.698596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.698605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.698685 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:12.698908 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.699655 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701178 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.701771 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.701862 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.702027 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.702068 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:12.702174 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.698248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:44709 (local address 127.27.68.65:56273)
0929 03:26:12.698501 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.698511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.698588 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.698737 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.698737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.698739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:12.698829 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.698831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.699734 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.699735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.701098 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.701101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.701863 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.701944 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.701952 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.702009 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.702058 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":88,"spinlock_wait_cycles":15616,"thread_start_us":96,"threads_started":1}
I20260929 03:26:12.702174 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.698248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:42341 (local address 127.27.68.65:45681)
0929 03:26:12.698415 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.698427 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:12.698521 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:12.698719 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:12.698720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:12.698722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:12.698833 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.698834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.699737 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.699737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.701098 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:12.701101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.701865 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.701866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:12.701944 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.701950 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:12.702009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:26:12.702058 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":110,"thread_start_us":99,"threads_started":1}
I20260929 03:26:12.702594 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbc23c3fad4c598e2b61d812a6beca" candidate_uuid: "03a2d9432bba4aebaa8261f106dd14b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5e17efec5416fa0c809c01e91ef15" is_pre_election: true
I20260929 03:26:12.702646 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbc23c3fad4c598e2b61d812a6beca" candidate_uuid: "03a2d9432bba4aebaa8261f106dd14b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24578260d2fa4daaa7884a6392f35696" is_pre_election: true
I20260929 03:26:12.702762 13825 raft_consensus.cc:2454] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a2d9432bba4aebaa8261f106dd14b9 in term 0.
I20260929 03:26:12.702795 13959 raft_consensus.cc:2454] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a2d9432bba4aebaa8261f106dd14b9 in term 0.
I20260929 03:26:12.703003 13625 leader_election.cc:302] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a2d9432bba4aebaa8261f106dd14b9, 6ea5e17efec5416fa0c809c01e91ef15; no voters:
I20260929 03:26:12.703151 14027 raft_consensus.cc:2781] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:12.703264 14027 raft_consensus.cc:482] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:12.703300 14027 raft_consensus.cc:3037] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:12.704021 14027 raft_consensus.cc:504] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.704187 14027 leader_election.cc:288] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [CANDIDATE]: Term 1 election: Requested vote from peers 6ea5e17efec5416fa0c809c01e91ef15 (127.27.68.66:44709), 24578260d2fa4daaa7884a6392f35696 (127.27.68.67:42341)
I20260929 03:26:12.704362 13959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbc23c3fad4c598e2b61d812a6beca" candidate_uuid: "03a2d9432bba4aebaa8261f106dd14b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24578260d2fa4daaa7884a6392f35696"
I20260929 03:26:12.704361 13825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfbc23c3fad4c598e2b61d812a6beca" candidate_uuid: "03a2d9432bba4aebaa8261f106dd14b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ea5e17efec5416fa0c809c01e91ef15"
I20260929 03:26:12.704456 13959 raft_consensus.cc:3037] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:12.704459 13825 raft_consensus.cc:3037] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:12.705192 13959 raft_consensus.cc:2454] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a2d9432bba4aebaa8261f106dd14b9 in term 1.
I20260929 03:26:12.705217 13825 raft_consensus.cc:2454] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a2d9432bba4aebaa8261f106dd14b9 in term 1.
I20260929 03:26:12.705408 13624 leader_election.cc:302] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a2d9432bba4aebaa8261f106dd14b9, 24578260d2fa4daaa7884a6392f35696; no voters:
I20260929 03:26:12.705490 14027 raft_consensus.cc:2781] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:12.705678 14027 raft_consensus.cc:686] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 1 LEADER]: Becoming Leader. State: Replica: 03a2d9432bba4aebaa8261f106dd14b9, State: Running, Role: LEADER
I20260929 03:26:12.705782 14027 consensus_queue.cc:237] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } }
I20260929 03:26:12.706633 13546 catalog_manager.cc:5495] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03a2d9432bba4aebaa8261f106dd14b9 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "03a2d9432bba4aebaa8261f106dd14b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a2d9432bba4aebaa8261f106dd14b9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 36055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 } health_report { overall_health: UNKNOWN } } }
I20260929 03:26:12.748364 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.745823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.27.68.65:36055)
0929 03:26:12.745889 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.745892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.745901 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:12.745940 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.745941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.745960 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.746039 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:12.746173 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.746646 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.747450 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.747693 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.747740 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.747937 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:12.748082 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:12.748163 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.748285 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.748297 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:26:12.751266 13825 raft_consensus.cc:1261] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 1 FOLLOWER]: Refusing update from remote peer 03a2d9432bba4aebaa8261f106dd14b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:12.751353 13959 raft_consensus.cc:1261] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 1 FOLLOWER]: Refusing update from remote peer 03a2d9432bba4aebaa8261f106dd14b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:12.751654 14030 consensus_queue.cc:1035] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ea5e17efec5416fa0c809c01e91ef15" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 44709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:12.751801 14027 consensus_queue.cc:1035] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24578260d2fa4daaa7884a6392f35696" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:12.754323 14034 mvcc.cc:205] Tried to move back new op lower bound from 7334512118786166784 to 7334512118603522048. Current Snapshot: MvccSnapshot[applied={T|T < 7334512118786166784}]
I20260929 03:26:12.755532 14035 mvcc.cc:205] Tried to move back new op lower bound from 7334512118786166784 to 7334512118603522048. Current Snapshot: MvccSnapshot[applied={T|T < 7334512118786166784 or (T in {7334512118786166784})}]
I20260929 03:26:12.757156 13546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:12.757272 13546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:12.758092 13546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fafef498e5d4ab9a14cd8b93a4e0474: Sending DeleteTablet for 3 replicas of tablet 4bfbc23c3fad4c598e2b61d812a6beca
I20260929 03:26:12.758455 13939 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:12 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:36662
I20260929 03:26:12.758462 13671 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:12 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:57678
I20260929 03:26:12.758517 13805 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfbc23c3fad4c598e2b61d812a6beca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:12 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:50028
I20260929 03:26:12.758740 14042 tablet_replica.cc:330] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9: stopping tablet replica
I20260929 03:26:12.758849 14044 tablet_replica.cc:330] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: stopping tablet replica
I20260929 03:26:12.758852 14042 raft_consensus.cc:2229] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:26:12.758962 14042 raft_consensus.cc:2258] T 4bfbc23c3fad4c598e2b61d812a6beca P 03a2d9432bba4aebaa8261f106dd14b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:12.758963 14044 raft_consensus.cc:2229] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:12.759079 14044 raft_consensus.cc:2258] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:12.759161 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13608
I20260929 03:26:12.763437 14043 tablet_replica.cc:330] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: stopping tablet replica
I20260929 03:26:12.763587 14043 raft_consensus.cc:2229] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:12.763666 14044 ts_tablet_manager.cc:1905] T 4bfbc23c3fad4c598e2b61d812a6beca P 6ea5e17efec5416fa0c809c01e91ef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:12.763712 14043 raft_consensus.cc:2258] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:12.764161 14043 ts_tablet_manager.cc:1905] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:26:12.764922 13531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:36055 (error 108)
I20260929 03:26:12.765182 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13742
I20260929 03:26:12.765761 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.765274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:36055 (local address 127.0.0.1:57686)
0929 03:26:12.765689 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.765725 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:36055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
W20260929 03:26:12.767822 13531 catalog_manager.cc:4584] TS 03a2d9432bba4aebaa8261f106dd14b9 (127.27.68.65:36055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bfbc23c3fad4c598e2b61d812a6beca: Network error: Client connection negotiation failed: client connection to 127.27.68.65:36055: connect: Connection refused (error 111)
I20260929 03:26:12.768921 14043 ts_tablet_manager.cc:1918] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:12.769006 14043 log.cc:1196] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/4bfbc23c3fad4c598e2b61d812a6beca
I20260929 03:26:12.769325 14043 ts_tablet_manager.cc:1939] T 4bfbc23c3fad4c598e2b61d812a6beca P 24578260d2fa4daaa7884a6392f35696: Deleting consensus metadata
I20260929 03:26:12.769786 13530 catalog_manager.cc:4841] TS 24578260d2fa4daaa7884a6392f35696 (127.27.68.67:42341): tablet 4bfbc23c3fad4c598e2b61d812a6beca (table test-table [id=4c528050b99b49239fee4412d08a3478]) successfully deleted
I20260929 03:26:12.770678 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.770556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:44709 (local address 127.0.0.1:50044)
0929 03:26:12.770613 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:12.770644 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.66:44709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:26:12.770671 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13876
I20260929 03:26:12.775321 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 13514
2026-09-29T03:26:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2105 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:12Z Disabled control of system clock
I20260929 03:26:12.803439 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:37447
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:42443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:12.885453 14049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:12.885689 14049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:12.885708 14049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:12.886926 14049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:12.886973 14049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:12.886986 14049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:12.886999 14049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:12.887012 14049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:12.888468 14049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:37447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:12.888728 14049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:12.889062 14049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:12.894011 14055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.894114 14054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:12.894013 14057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:12.894212 14049 server_base.cc:1034] running on GCE node
I20260929 03:26:12.894423 14049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:12.894716 14049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:12.895864 14049 hybrid_clock.cc:648] HybridClock initialized: now 1790652372895850 us; error 47 us; skew 500 ppm
I20260929 03:26:12.897221 14049 webserver.cc:460] Webserver started at http://127.27.68.126:41939/ using document root <none> and password file <none>
I20260929 03:26:12.897506 14049 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:12.897560 14049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:12.897656 14049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:12.898701 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "82c4d2dde42c44b0b8667103e1dad161"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "b21cfdf8ca540ab7c5eb4198eedbf390"
server_key_iv: "130e6d755b96a2e629327b869cebeaee"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.898996 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "82c4d2dde42c44b0b8667103e1dad161"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "b21cfdf8ca540ab7c5eb4198eedbf390"
server_key_iv: "130e6d755b96a2e629327b869cebeaee"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.900429 14049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:12.901134 14063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:12.901319 14049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:12.901409 14049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "82c4d2dde42c44b0b8667103e1dad161"
format_stamp: "Formatted at 2026-09-29 03:26:12 on dist-test-slave-fzwk"
server_key: "b21cfdf8ca540ab7c5eb4198eedbf390"
server_key_iv: "130e6d755b96a2e629327b869cebeaee"
server_key_version: "encryptionkey@0"
I20260929 03:26:12.901465 14049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:12.922720 14049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:12.923017 14049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:12.923130 14049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:12.927268 14049 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:37447
I20260929 03:26:12.927323 14115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:37447 every 8 connection(s)
I20260929 03:26:12.927681 14049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:12.927939 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14049
I20260929 03:26:12.928058 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:12.928267 27921 external_mini_cluster.cc:1428] Setting key 9836d7d2e07e209defc16bb2c4f1d9ba
I20260929 03:26:12.928833 14116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:12.931550 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: Bootstrap starting.
I20260929 03:26:12.932160 14116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:12.932510 14116 log.cc:824] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:12.933353 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: No bootstrap required, opened a new log
I20260929 03:26:12.933832 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:12.928829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.27.68.126:37447)
0929 03:26:12.929066 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:12.929074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:12.929100 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:26:12.929528 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:12.929541 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:12.929602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:12.929753 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:12.929986 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.930862 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.931805 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:12.932224 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:12.932365 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:12.933112 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:12.933116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:12.933116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:12.933323 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:12.933365 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:12.933380 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:12.933450 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:12.933542 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:26:12.933624 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260929 03:26:12.935478 14116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } }
I20260929 03:26:12.935623 14116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:12.935647 14116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c4d2dde42c44b0b8667103e1dad161, State: Initialized, Role: FOLLOWER
I20260929 03:26:12.935796 14116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } }
I20260929 03:26:12.935858 14116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:12.935900 14116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:12.935949 14116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:12.936520 14116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } }
I20260929 03:26:12.936642 14116 leader_election.cc:302] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c4d2dde42c44b0b8667103e1dad161; no voters:
I20260929 03:26:12.936837 14116 leader_election.cc:288] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:12.936909 14121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:12.937093 14121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [term 1 LEADER]: Becoming Leader. State: Replica: 82c4d2dde42c44b0b8667103e1dad161, State: Running, Role: LEADER
I20260929 03:26:12.937152 14116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:12.937206 14121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } }
I20260929 03:26:12.937604 14122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c4d2dde42c44b0b8667103e1dad161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } } }
I20260929 03:26:12.937625 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c4d2dde42c44b0b8667103e1dad161. Latest consensus state: current_term: 1 leader_uuid: "82c4d2dde42c44b0b8667103e1dad161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c4d2dde42c44b0b8667103e1dad161" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 37447 } } }
I20260929 03:26:12.937742 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:12.937726 14122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:12.937981 14130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:12.938498 14130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:12.940124 14130 catalog_manager.cc:1347] Generated new cluster ID: acca9424c4b4486d90e9d31484bbafa8
I20260929 03:26:12.940178 14130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:12.943854 14130 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:12.944577 14130 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:12.959533 14130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: Generated new TSK 0
I20260929 03:26:12.959772 14130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:12.968204 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--builtin_ntp_servers=127.27.68.84:42443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:13.057896 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:13.058151 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:13.058174 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:13.059413 14140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:13.059455 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:13.059481 14140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:13.060925 14140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:13.061192 14140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:13.061524 14140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:13.067576 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:13.067469 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:13.067730 14140 server_base.cc:1034] running on GCE node
W20260929 03:26:13.067476 14148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:13.068056 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:13.068356 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:13.069525 14140 hybrid_clock.cc:648] HybridClock initialized: now 1790652373069509 us; error 34 us; skew 500 ppm
I20260929 03:26:13.070995 14140 webserver.cc:460] Webserver started at http://127.27.68.65:45995/ using document root <none> and password file <none>
I20260929 03:26:13.071354 14140 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:13.071408 14140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:13.071565 14140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:13.072702 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "ee8771069c98459d8dd4e6954e2773d9"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "9a65e551cd6c3ddc0d0804b31a749f9d"
server_key_iv: "f1628c1f19e188a03935df080da4948c"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.073081 14140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "ee8771069c98459d8dd4e6954e2773d9"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "9a65e551cd6c3ddc0d0804b31a749f9d"
server_key_iv: "f1628c1f19e188a03935df080da4948c"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.074594 14140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:13.075485 14154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.075703 14140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:13.075809 14140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "ee8771069c98459d8dd4e6954e2773d9"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "9a65e551cd6c3ddc0d0804b31a749f9d"
server_key_iv: "f1628c1f19e188a03935df080da4948c"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.075886 14140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:13.088691 14140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:13.088990 14140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:13.089108 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:13.089313 14140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:13.089701 14140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:13.089741 14140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.089790 14140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:13.089821 14140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.095887 14140 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:45315
I20260929 03:26:13.095968 14267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:45315 every 8 connection(s)
I20260929 03:26:13.096349 14140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:26:13.101650 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.097162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:44701 (local address 127.27.68.126:37447)
0929 03:26:13.097226 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.097228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.097499 (+ 271us) server_negotiation.cc:407] Connection header received
0929 03:26:13.097994 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.097994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.098008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.098073 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:13.098764 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.099351 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.100332 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.100523 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.100582 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.101385 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:13.101386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:13.101387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:13.101392 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:13.101406 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:13.101410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:13.101453 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.101540 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.101584 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:26:13.101869 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.097128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37447 (local address 127.27.68.65:44701)
0929 03:26:13.097334 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.097378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.097905 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.098143 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.098146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.098156 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:13.098681 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.098685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.099437 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.099437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.100248 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.100251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.100609 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.100610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.100729 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.101246 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:13.101247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:13.101277 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:13.101302 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:13.101467 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:13.101471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:13.101473 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.101527 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.101620 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":114,"threads_started":1}
I20260929 03:26:13.102437 14268 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37447
I20260929 03:26:13.102556 14268 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:13.102747 14268 heartbeater.cc:511] Master 127.27.68.126:37447 requested a full tablet report, sending...
I20260929 03:26:13.103381 14080 ts_manager.cc:194] Registered new tserver with Master: ee8771069c98459d8dd4e6954e2773d9 (127.27.68.65:45315)
I20260929 03:26:13.103962 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14140
I20260929 03:26:13.104076 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:13.104050 14080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.65:44701
I20260929 03:26:13.104301 27921 external_mini_cluster.cc:1428] Setting key b04fcf7be74617f627222e99305eb5b7
I20260929 03:26:13.105305 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--builtin_ntp_servers=127.27.68.84:42443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:13.188169 14271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:13.188412 14271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:13.188448 14271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:13.189606 14271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:13.189675 14271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:13.189715 14271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:13.191113 14271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:13.191428 14271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:13.191751 14271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:13.196867 14279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:13.196864 14277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:13.196862 14276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:13.197017 14271 server_base.cc:1034] running on GCE node
I20260929 03:26:13.197255 14271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:13.197458 14271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:13.198596 14271 hybrid_clock.cc:648] HybridClock initialized: now 1790652373198575 us; error 37 us; skew 500 ppm
I20260929 03:26:13.199903 14271 webserver.cc:460] Webserver started at http://127.27.68.66:34353/ using document root <none> and password file <none>
I20260929 03:26:13.200156 14271 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:13.200204 14271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:13.200382 14271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:13.201483 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "00dd195969b143b5856d81e54f3010b6"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "500cf29d37823d34a0a4f8aa0de97e1e"
server_key_iv: "69327e7d7ced51fc6cdb7452e23ffff7"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.201845 14271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "00dd195969b143b5856d81e54f3010b6"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "500cf29d37823d34a0a4f8aa0de97e1e"
server_key_iv: "69327e7d7ced51fc6cdb7452e23ffff7"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.203166 14271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:13.203976 14285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.204169 14271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:13.204263 14271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "00dd195969b143b5856d81e54f3010b6"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "500cf29d37823d34a0a4f8aa0de97e1e"
server_key_iv: "69327e7d7ced51fc6cdb7452e23ffff7"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.204371 14271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:13.222339 14271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:13.222613 14271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:13.222714 14271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:13.222887 14271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:13.223285 14271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:13.223325 14271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.223377 14271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:13.223407 14271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.229306 14271 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:39663
I20260929 03:26:13.229350 14398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:39663 every 8 connection(s)
I20260929 03:26:13.229717 14271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:26:13.229853 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14271
I20260929 03:26:13.229974 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:13.230226 27921 external_mini_cluster.cc:1428] Setting key 7a26d8b71da8171e8a8ed28027c35434
I20260929 03:26:13.231364 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--builtin_ntp_servers=127.27.68.84:42443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:26:13.235687 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.230563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:32931 (local address 127.27.68.126:37447)
0929 03:26:13.230618 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.230621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.230883 (+ 262us) server_negotiation.cc:407] Connection header received
0929 03:26:13.231623 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.231624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.231635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.231680 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:13.232593 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.233076 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.234113 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.234308 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.234392 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.235363 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:13.235365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:13.235365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:13.235371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:13.235392 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:13.235397 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:13.235458 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.235571 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.235603 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:13.235903 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.230534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37447 (local address 127.27.68.66:32931)
0929 03:26:13.230723 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.230766 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.231520 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.231757 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.231761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.231772 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:13.232497 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.232504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.233180 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.233181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.233997 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.233999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.234404 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.234405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.234552 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.235162 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:13.235162 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:13.235191 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:13.235255 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:13.235458 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:13.235464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:13.235468 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.235573 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.235678 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 03:26:13.236377 14399 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37447
I20260929 03:26:13.236491 14399 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:13.236737 14399 heartbeater.cc:511] Master 127.27.68.126:37447 requested a full tablet report, sending...
I20260929 03:26:13.237181 14080 ts_manager.cc:194] Registered new tserver with Master: 00dd195969b143b5856d81e54f3010b6 (127.27.68.66:39663)
I20260929 03:26:13.237545 14080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.66:32931
W20260929 03:26:13.331287 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:13.331529 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:13.331552 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:13.332801 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:13.332847 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:13.332875 14402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:13.334303 14402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:42443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:13.334596 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:13.334903 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:13.340106 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:13.340133 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:13.340113 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:13.340371 14402 server_base.cc:1034] running on GCE node
I20260929 03:26:13.340593 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:13.340842 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:13.342010 14402 hybrid_clock.cc:648] HybridClock initialized: now 1790652373341995 us; error 44 us; skew 500 ppm
I20260929 03:26:13.343324 14402 webserver.cc:460] Webserver started at http://127.27.68.67:40547/ using document root <none> and password file <none>
I20260929 03:26:13.343598 14402 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:13.343670 14402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:13.343829 14402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:13.344923 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "da6a5d23dac640e0a88f1b58607e94ad"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "1584e2610e4c4a624921d5f8ae322f4b"
server_key_iv: "9d68d6330092181f339891aad50a7ef3"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.345286 14402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "da6a5d23dac640e0a88f1b58607e94ad"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "1584e2610e4c4a624921d5f8ae322f4b"
server_key_iv: "9d68d6330092181f339891aad50a7ef3"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.346532 14402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:13.347200 14416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.347404 14402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:13.347502 14402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "da6a5d23dac640e0a88f1b58607e94ad"
format_stamp: "Formatted at 2026-09-29 03:26:13 on dist-test-slave-fzwk"
server_key: "1584e2610e4c4a624921d5f8ae322f4b"
server_key_iv: "9d68d6330092181f339891aad50a7ef3"
server_key_version: "encryptionkey@0"
I20260929 03:26:13.347589 14402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:13.373730 14402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:13.374092 14402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:13.374254 14402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:13.374516 14402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:13.374873 14402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:13.374930 14402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.374974 14402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:13.375000 14402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:13.381155 14402 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:37527
I20260929 03:26:13.381197 14529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:37527 every 8 connection(s)
I20260929 03:26:13.381564 14402 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:26:13.386869 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.382268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:44701 (local address 127.27.68.126:37447)
0929 03:26:13.382331 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.382333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.382691 (+ 358us) server_negotiation.cc:407] Connection header received
0929 03:26:13.383172 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.383173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.383184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.383348 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:13.384061 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.384561 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.385573 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.385758 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.385814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.386634 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:13.386635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:13.386635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:13.386640 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:13.386655 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:13.386659 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:13.386703 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.386784 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.386807 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:26:13.387048 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14402
I20260929 03:26:13.387154 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:13.387161 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.382277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:37447 (local address 127.27.68.67:44701)
0929 03:26:13.382543 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.382586 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.383087 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.383455 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.383459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.383469 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:13.383977 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.383981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.384648 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.384649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.385487 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.385490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.385840 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.385841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.385956 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.386488 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:13.386488 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:13.386519 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:13.386549 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:13.386715 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:13.386719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:13.386722 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.386774 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.386905 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":148,"threads_started":1}
I20260929 03:26:13.387426 27921 external_mini_cluster.cc:1428] Setting key 3faec84b24666048630bffd284180561
I20260929 03:26:13.387635 14530 heartbeater.cc:347] Connected to a master server at 127.27.68.126:37447
I20260929 03:26:13.387732 14530 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:13.387935 14530 heartbeater.cc:511] Master 127.27.68.126:37447 requested a full tablet report, sending...
I20260929 03:26:13.388357 14080 ts_manager.cc:194] Registered new tserver with Master: da6a5d23dac640e0a88f1b58607e94ad (127.27.68.67:37527)
I20260929 03:26:13.388713 14080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.67:44701
I20260929 03:26:13.399549 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:13.404052 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.401063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.27.68.126:37447)
0929 03:26:13.401138 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.401140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.401252 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:26:13.401297 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.401297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.401307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.401353 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:13.401489 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.401968 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.402696 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.402879 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.402930 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.403709 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:13.403710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:13.403711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:13.403715 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:13.403728 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:13.403732 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:13.403808 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.403896 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.403928 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:26:13.405843 14080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:13.407644 14080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:13.414274 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.409869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:45315 (local address 127.0.0.1:37162)
0929 03:26:13.410209 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.410219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.410288 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.410473 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.410476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.410478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:13.410565 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.410567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.411735 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.411736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.413006 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.413009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.413966 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.414047 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.414063 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.414134 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.414188 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":59,"threads_started":1}
I20260929 03:26:13.414299 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.409985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.27.68.65:45315)
0929 03:26:13.410182 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.410187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.410273 (+ 86us) server_negotiation.cc:407] Connection header received
0929 03:26:13.410366 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.410369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.410386 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.410467 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:13.410654 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.411652 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413105 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.413855 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.414033 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.414160 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.414202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260929 03:26:13.414454 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.410080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:37527 (local address 127.0.0.1:47212)
0929 03:26:13.410284 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.410293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.410334 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.410690 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.410691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.410694 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:13.410744 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.410746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.412073 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.412074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.413121 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.413123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.414086 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.414088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.414177 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.414183 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.414236 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.414278 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260929 03:26:13.414918 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.410283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.27.68.67:37527)
0929 03:26:13.410446 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.410451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.410462 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:13.410503 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.410505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.410520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.410605 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:13.410831 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.411962 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413204 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.413866 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413932 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.414029 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.414325 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.414824 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260929 03:26:13.415181 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.409831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39663 (local address 127.0.0.1:57792)
0929 03:26:13.410049 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.410088 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.410162 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.410597 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.410598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.410600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:13.410637 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.410638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.413204 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.413849 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.413851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.414925 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.414926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.414996 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.415002 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.415066 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.415106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
I20260929 03:26:13.415248 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.409924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.27.68.66:39663)
0929 03:26:13.410454 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.410459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.410468 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:13.410504 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.410506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.410520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.410600 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:13.410774 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.411973 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.413944 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.414833 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.414907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.414988 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.415092 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.415132 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":96,"threads_started":1}
I20260929 03:26:13.415160 14202 tablet_service.cc:1467] Processing CreateTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 (DEFAULT_TABLE table=test-table [id=4ce712c77c3447b1a6d963b24df67356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:13.415601 14202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77bc4e3dbdd9416ba83ec3e47d126512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:13.415917 14333 tablet_service.cc:1467] Processing CreateTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 (DEFAULT_TABLE table=test-table [id=4ce712c77c3447b1a6d963b24df67356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:13.416199 14333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77bc4e3dbdd9416ba83ec3e47d126512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:13.416286 14464 tablet_service.cc:1467] Processing CreateTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 (DEFAULT_TABLE table=test-table [id=4ce712c77c3447b1a6d963b24df67356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:13.416600 14464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77bc4e3dbdd9416ba83ec3e47d126512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:13.418747 14544 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Bootstrap starting.
I20260929 03:26:13.419075 14545 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: Bootstrap starting.
I20260929 03:26:13.419431 14544 tablet_bootstrap.cc:654] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:13.419730 14545 tablet_bootstrap.cc:654] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:13.419759 14544 log.cc:824] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:13.420042 14545 log.cc:824] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:13.420225 14547 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Bootstrap starting.
I20260929 03:26:13.420606 14544 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: No bootstrap required, opened a new log
I20260929 03:26:13.420663 14544 ts_tablet_manager.cc:1397] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:13.420783 14547 tablet_bootstrap.cc:654] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:13.421052 14547 log.cc:824] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:13.421746 14547 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: No bootstrap required, opened a new log
I20260929 03:26:13.421833 14547 ts_tablet_manager.cc:1397] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:13.422748 14544 raft_consensus.cc:348] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.422883 14544 raft_consensus.cc:374] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:13.422906 14544 raft_consensus.cc:729] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00dd195969b143b5856d81e54f3010b6, State: Initialized, Role: FOLLOWER
I20260929 03:26:13.423032 14544 consensus_queue.cc:260] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.423333 14544 ts_tablet_manager.cc:1428] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:26:13.423373 14399 heartbeater.cc:503] Master 127.27.68.126:37447 was elected leader, sending a full tablet report...
I20260929 03:26:13.423882 14545 tablet_bootstrap.cc:492] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: No bootstrap required, opened a new log
I20260929 03:26:13.423967 14545 ts_tablet_manager.cc:1397] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:26:13.424341 14547 raft_consensus.cc:348] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.424497 14547 raft_consensus.cc:374] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:13.424564 14547 raft_consensus.cc:729] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6a5d23dac640e0a88f1b58607e94ad, State: Initialized, Role: FOLLOWER
I20260929 03:26:13.424682 14547 consensus_queue.cc:260] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.424980 14547 ts_tablet_manager.cc:1428] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:26:13.425024 14530 heartbeater.cc:503] Master 127.27.68.126:37447 was elected leader, sending a full tablet report...
I20260929 03:26:13.426550 14545 raft_consensus.cc:348] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.426719 14545 raft_consensus.cc:374] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:13.426765 14545 raft_consensus.cc:729] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8771069c98459d8dd4e6954e2773d9, State: Initialized, Role: FOLLOWER
I20260929 03:26:13.426887 14545 consensus_queue.cc:260] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.427129 14545 ts_tablet_manager.cc:1428] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:26:13.427209 14268 heartbeater.cc:503] Master 127.27.68.126:37447 was elected leader, sending a full tablet report...
I20260929 03:26:13.428074 14552 raft_consensus.cc:482] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:13.428177 14552 raft_consensus.cc:504] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.428474 14552 leader_election.cc:288] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00dd195969b143b5856d81e54f3010b6 (127.27.68.66:39663), da6a5d23dac640e0a88f1b58607e94ad (127.27.68.67:37527)
I20260929 03:26:13.432142 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.428784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:33785 (local address 127.27.68.67:37527)
0929 03:26:13.428827 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.428829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.428943 (+ 114us) server_negotiation.cc:407] Connection header received
0929 03:26:13.428993 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.428994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.429003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.429071 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:13.429167 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.429833 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431273 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.431772 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431859 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.431868 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.432018 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.432060 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:13.432163 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.428637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:49383 (local address 127.27.68.66:39663)
0929 03:26:13.428781 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.428784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.428791 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:26:13.428814 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.428815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.428826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.428876 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:13.429024 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.429776 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431289 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.431773 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431860 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.431869 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.432052 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.432101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 03:26:13.432163 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.428719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:37527 (local address 127.27.68.65:33785)
0929 03:26:13.428885 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.428894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.428949 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.429073 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.429074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.429075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:13.429128 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.429129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.429903 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.429904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.431140 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.431144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.431861 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.431937 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.431943 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.431993 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.432052 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":6,"thread_start_us":85,"threads_started":1}
I20260929 03:26:13.432170 14270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.428588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:39663 (local address 127.27.68.65:49383)
0929 03:26:13.428647 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.428660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:13.428699 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:13.428877 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:13.428878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:13.428880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:13.428944 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.428946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.429855 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.429856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.431140 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:13.431144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.431860 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.431861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:13.431953 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.431956 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:13.432030 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:26:13.432088 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":38}
I20260929 03:26:13.432749 14484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77bc4e3dbdd9416ba83ec3e47d126512" candidate_uuid: "ee8771069c98459d8dd4e6954e2773d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6a5d23dac640e0a88f1b58607e94ad" is_pre_election: true
I20260929 03:26:13.432780 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77bc4e3dbdd9416ba83ec3e47d126512" candidate_uuid: "ee8771069c98459d8dd4e6954e2773d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00dd195969b143b5856d81e54f3010b6" is_pre_election: true
I20260929 03:26:13.432896 14484 raft_consensus.cc:2454] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8771069c98459d8dd4e6954e2773d9 in term 0.
I20260929 03:26:13.432914 14353 raft_consensus.cc:2454] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8771069c98459d8dd4e6954e2773d9 in term 0.
I20260929 03:26:13.433133 14157 leader_election.cc:302] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00dd195969b143b5856d81e54f3010b6, ee8771069c98459d8dd4e6954e2773d9; no voters:
I20260929 03:26:13.433280 14552 raft_consensus.cc:2781] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:13.433352 14552 raft_consensus.cc:482] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:13.433370 14552 raft_consensus.cc:3037] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:13.434113 14552 raft_consensus.cc:504] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.434245 14552 leader_election.cc:288] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [CANDIDATE]: Term 1 election: Requested vote from peers 00dd195969b143b5856d81e54f3010b6 (127.27.68.66:39663), da6a5d23dac640e0a88f1b58607e94ad (127.27.68.67:37527)
I20260929 03:26:13.434412 14353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77bc4e3dbdd9416ba83ec3e47d126512" candidate_uuid: "ee8771069c98459d8dd4e6954e2773d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00dd195969b143b5856d81e54f3010b6"
I20260929 03:26:13.434473 14484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77bc4e3dbdd9416ba83ec3e47d126512" candidate_uuid: "ee8771069c98459d8dd4e6954e2773d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6a5d23dac640e0a88f1b58607e94ad"
I20260929 03:26:13.434514 14353 raft_consensus.cc:3037] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:13.434558 14484 raft_consensus.cc:3037] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:13.435338 14353 raft_consensus.cc:2454] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8771069c98459d8dd4e6954e2773d9 in term 1.
I20260929 03:26:13.435336 14484 raft_consensus.cc:2454] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8771069c98459d8dd4e6954e2773d9 in term 1.
I20260929 03:26:13.435501 14157 leader_election.cc:302] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00dd195969b143b5856d81e54f3010b6, ee8771069c98459d8dd4e6954e2773d9; no voters:
I20260929 03:26:13.435592 14552 raft_consensus.cc:2781] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:13.435743 14552 raft_consensus.cc:686] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [term 1 LEADER]: Becoming Leader. State: Replica: ee8771069c98459d8dd4e6954e2773d9, State: Running, Role: LEADER
I20260929 03:26:13.435844 14552 consensus_queue.cc:237] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } }
I20260929 03:26:13.436640 14080 catalog_manager.cc:5495] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee8771069c98459d8dd4e6954e2773d9 (127.27.68.65). New cstate: current_term: 1 leader_uuid: "ee8771069c98459d8dd4e6954e2773d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8771069c98459d8dd4e6954e2773d9" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 45315 } health_report { overall_health: HEALTHY } } }
I20260929 03:26:13.442209 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.439697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.27.68.65:45315)
0929 03:26:13.439758 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:13.439761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:13.439769 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:26:13.439869 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:13.439870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:13.439889 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:13.439998 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:13.440142 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.440612 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.441381 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:13.441603 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:13.441682 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:13.441827 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:13.441930 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:13.442040 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:13.442091 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:26:13.442102 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:26:13.444000 14484 raft_consensus.cc:1261] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 1 FOLLOWER]: Refusing update from remote peer ee8771069c98459d8dd4e6954e2773d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:13.444002 14353 raft_consensus.cc:1261] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 1 FOLLOWER]: Refusing update from remote peer ee8771069c98459d8dd4e6954e2773d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:13.444366 14552 consensus_queue.cc:1035] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00dd195969b143b5856d81e54f3010b6" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:13.444504 14554 consensus_queue.cc:1035] T 77bc4e3dbdd9416ba83ec3e47d126512 P ee8771069c98459d8dd4e6954e2773d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6a5d23dac640e0a88f1b58607e94ad" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 37527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:13.447719 14559 mvcc.cc:205] Tried to move back new op lower bound from 7334512121624027136 to 7334512121593864192. Current Snapshot: MvccSnapshot[applied={T|T < 7334512121624027136}]
I20260929 03:26:13.447850 14558 mvcc.cc:205] Tried to move back new op lower bound from 7334512121624027136 to 7334512121593864192. Current Snapshot: MvccSnapshot[applied={T|T < 7334512121624027136 or (T in {7334512121624027136})}]
I20260929 03:26:13.450449 14080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:13.450541 14080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:13.451391 14080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82c4d2dde42c44b0b8667103e1dad161: Sending DeleteTablet for 3 replicas of tablet 77bc4e3dbdd9416ba83ec3e47d126512
I20260929 03:26:13.451713 14202 tablet_service.cc:1514] Processing DeleteTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:13 UTC) from {username='slave'} at 127.0.0.1:37162
I20260929 03:26:13.451753 14333 tablet_service.cc:1514] Processing DeleteTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:13 UTC) from {username='slave'} at 127.0.0.1:57792
I20260929 03:26:13.451988 14464 tablet_service.cc:1514] Processing DeleteTablet for tablet 77bc4e3dbdd9416ba83ec3e47d126512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:13 UTC) from {username='slave'} at 127.0.0.1:47212
I20260929 03:26:13.452174 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14140
I20260929 03:26:13.452266 14568 tablet_replica.cc:330] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: stopping tablet replica
I20260929 03:26:13.452406 14568 raft_consensus.cc:2229] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:13.452592 14568 raft_consensus.cc:2258] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:13.452230 14567 tablet_replica.cc:330] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: stopping tablet replica
I20260929 03:26:13.452925 14567 raft_consensus.cc:2229] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:13.453037 14567 raft_consensus.cc:2258] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:13.453293 14568 ts_tablet_manager.cc:1905] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:13.453572 14567 ts_tablet_manager.cc:1905] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:13.455307 14568 ts_tablet_manager.cc:1918] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:13.455338 14567 ts_tablet_manager.cc:1918] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:13.455389 14568 log.cc:1196] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/77bc4e3dbdd9416ba83ec3e47d126512
I20260929 03:26:13.455415 14567 log.cc:1196] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/77bc4e3dbdd9416ba83ec3e47d126512
I20260929 03:26:13.455653 14568 ts_tablet_manager.cc:1939] T 77bc4e3dbdd9416ba83ec3e47d126512 P da6a5d23dac640e0a88f1b58607e94ad: Deleting consensus metadata
I20260929 03:26:13.455667 14567 ts_tablet_manager.cc:1939] T 77bc4e3dbdd9416ba83ec3e47d126512 P 00dd195969b143b5856d81e54f3010b6: Deleting consensus metadata
I20260929 03:26:13.456060 14066 catalog_manager.cc:4841] TS da6a5d23dac640e0a88f1b58607e94ad (127.27.68.67:37527): tablet 77bc4e3dbdd9416ba83ec3e47d126512 (table test-table [id=4ce712c77c3447b1a6d963b24df67356]) successfully deleted
I20260929 03:26:13.456187 14066 catalog_manager.cc:4841] TS 00dd195969b143b5856d81e54f3010b6 (127.27.68.66:39663): tablet 77bc4e3dbdd9416ba83ec3e47d126512 (table test-table [id=4ce712c77c3447b1a6d963b24df67356]) successfully deleted
W20260929 03:26:13.458231 14065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:45315 (error 108)
I20260929 03:26:13.458712 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:13.458572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:45315 (local address 127.0.0.1:37186)
0929 03:26:13.458630 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:13.458674 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:45315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:26:13.458735 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14271
W20260929 03:26:13.458854 14065 catalog_manager.cc:4584] TS ee8771069c98459d8dd4e6954e2773d9 (127.27.68.65:45315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77bc4e3dbdd9416ba83ec3e47d126512: Network error: Client connection negotiation failed: client connection to 127.27.68.65:45315: connect: Connection refused (error 111)
I20260929 03:26:13.463582 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14402
I20260929 03:26:13.468475 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14049
2026-09-29T03:26:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (691 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:13 dist-test-slave-fzwk krb5kdc[14572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:13 dist-test-slave-fzwk krb5kdc[14572](info): set up 2 sockets
Sep 29 03:26:13 dist-test-slave-fzwk krb5kdc[14572](info): commencing operation
krb5kdc: starting...
W20260929 03:26:14.499554 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:14 dist-test-slave-fzwk krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:14.622629 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41237
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:14.704872 14585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:14.705139 14585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:14.705179 14585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:14.706387 14585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:14.706466 14585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:14.706492 14585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:14.706514 14585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:14.706552 14585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:14.708115 14585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:41237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:14.708410 14585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:14.708652 14585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:14.713524 14591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:14.713666 14585 server_base.cc:1034] running on GCE node
W20260929 03:26:14.713531 14590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:14.713799 14593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:14.714047 14585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:14.714350 14585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:14.715523 14585 hybrid_clock.cc:648] HybridClock initialized: now 1790652374715502 us; error 50 us; skew 500 ppm
Sep 29 03:26:14 dist-test-slave-fzwk krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:14.724404 14585 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:14.724939 14585 webserver.cc:460] Webserver started at http://127.27.68.126:44835/ using document root <none> and password file <none>
I20260929 03:26:14.725203 14585 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:14.725281 14585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:14.725443 14585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:14.726593 14585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "d4866504264e49a89e6cb7e3c6e883d6"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "562183ede128e111066357731c14b0ad"
server_key_iv: "f3f48fb5885f6eeb122bc4fd8cd069fc"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.726992 14585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "d4866504264e49a89e6cb7e3c6e883d6"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "562183ede128e111066357731c14b0ad"
server_key_iv: "f3f48fb5885f6eeb122bc4fd8cd069fc"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.728574 14585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:14.729408 14600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:14.729609 14585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:14.729727 14585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "d4866504264e49a89e6cb7e3c6e883d6"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "562183ede128e111066357731c14b0ad"
server_key_iv: "f3f48fb5885f6eeb122bc4fd8cd069fc"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.729816 14585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:14.744392 14585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:14.746382 14585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:14.746569 14585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:14.750663 14585 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:41237
I20260929 03:26:14.750700 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:41237 every 8 connection(s)
I20260929 03:26:14.751094 14585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:14.752103 14653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:14.754528 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: Bootstrap starting.
I20260929 03:26:14.755162 14653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:14.755487 14653 log.cc:824] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:14.756197 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: No bootstrap required, opened a new log
I20260929 03:26:14.757939 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14585
I20260929 03:26:14.757851 14653 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } }
I20260929 03:26:14.758046 14653 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:14.758074 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:14.758098 14653 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4866504264e49a89e6cb7e3c6e883d6, State: Initialized, Role: FOLLOWER
I20260929 03:26:14.758276 27921 external_mini_cluster.cc:1428] Setting key 7c0ba9c7cb02cb3b2c497d59363e9a87
I20260929 03:26:14.758275 14653 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } }
I20260929 03:26:14.758355 14653 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:14.758400 14653 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:14.758453 14653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:14.759711 14653 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } }
I20260929 03:26:14.759811 14653 leader_election.cc:302] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4866504264e49a89e6cb7e3c6e883d6; no voters:
I20260929 03:26:14.760025 14653 leader_election.cc:288] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:14.760082 14658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:14.760223 14658 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [term 1 LEADER]: Becoming Leader. State: Replica: d4866504264e49a89e6cb7e3c6e883d6, State: Running, Role: LEADER
I20260929 03:26:14.760306 14653 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:14.760335 14658 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } }
I20260929 03:26:14.760684 14659 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4866504264e49a89e6cb7e3c6e883d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } } }
I20260929 03:26:14.760775 14659 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:14.761018 14660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4866504264e49a89e6cb7e3c6e883d6. Latest consensus state: current_term: 1 leader_uuid: "d4866504264e49a89e6cb7e3c6e883d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4866504264e49a89e6cb7e3c6e883d6" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41237 } } }
I20260929 03:26:14.761150 14660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:14.761313 14665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:14.761853 14665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:14.763424 14665 catalog_manager.cc:1347] Generated new cluster ID: c4cb696b53fb4bbe8990d683b0caff5f
I20260929 03:26:14.763485 14665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:14 dist-test-slave-fzwk krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:14.771122 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:14.758768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.27.68.126:41237)
0929 03:26:14.759001 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:14.759014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:14.759069 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:26:14.759265 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:14.759275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:14.759354 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:14.759507 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:14.760299 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.761809 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.762768 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.763413 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.763522 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:14.766322 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:14.766327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:14.766327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:14.766340 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:14.769074 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:14.769474 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:14.769477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:14.769478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:14.769603 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:14.769778 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:14.769779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:14.769779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:14.770088 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:14.770216 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:14.770719 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 03:26:14.770862 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
I20260929 03:26:14.776561 14665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:14.777298 14665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:14.791973 14665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: Generated new TSK 0
I20260929 03:26:14.792330 14665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.68.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
Entry for principal kudu/127.27.68.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab.
I20260929 03:26:14.825898 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:14.910214 14679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:14.910498 14679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:14.910531 14679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:14.911811 14679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:14.911870 14679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:14.911926 14679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:14.913480 14679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.65.keytab
--principal=kudu/127.27.68.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:14.913748 14679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:14.914001 14679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:14.919072 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:14.919070 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:14.919045 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:14.919356 14679 server_base.cc:1034] running on GCE node
I20260929 03:26:14.919580 14679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:14.919857 14679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:14.921030 14679 hybrid_clock.cc:648] HybridClock initialized: now 1790652374921023 us; error 41 us; skew 500 ppm
Sep 29 03:26:14 dist-test-slave-fzwk krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:14.923666 14679 init.cc:377] Logged in from keytab as kudu/127.27.68.65@KRBTEST.COM (short username kudu)
I20260929 03:26:14.924153 14679 webserver.cc:460] Webserver started at http://127.27.68.65:35917/ using document root <none> and password file <none>
I20260929 03:26:14.924397 14679 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:14.924467 14679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:14.924638 14679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:14.925776 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "4b737e492edf4a0789b4387ebc0a3483"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "52205e8601f345b8bb6d984508e594b5"
server_key_iv: "64980dbc6847045b447caf797c9ed918"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.926151 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "4b737e492edf4a0789b4387ebc0a3483"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "52205e8601f345b8bb6d984508e594b5"
server_key_iv: "64980dbc6847045b447caf797c9ed918"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.927589 14679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:14.928360 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:14.928639 14679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:14.928750 14679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "4b737e492edf4a0789b4387ebc0a3483"
format_stamp: "Formatted at 2026-09-29 03:26:14 on dist-test-slave-fzwk"
server_key: "52205e8601f345b8bb6d984508e594b5"
server_key_iv: "64980dbc6847045b447caf797c9ed918"
server_key_version: "encryptionkey@0"
I20260929 03:26:14.928828 14679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:14.942936 14679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:14.944969 14679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:14.945147 14679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:14.945391 14679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:14.945717 14679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:14.945772 14679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:14.945818 14679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:14.945850 14679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:14.952040 14679 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:38339
I20260929 03:26:14.952112 14807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:38339 every 8 connection(s)
I20260929 03:26:14.952447 14679 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:26:14 dist-test-slave-fzwk krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.65@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:14.961315 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14679
I20260929 03:26:14.961465 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:14.961724 27921 external_mini_cluster.cc:1428] Setting key 780a74ac2bd96f929147b26f22cfbe9f
I20260929 03:26:14.962200 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:14.953227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:58009 (local address 127.27.68.126:41237)
0929 03:26:14.953274 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:14.953276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:14.953505 (+ 229us) server_negotiation.cc:407] Connection header received
0929 03:26:14.953993 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:14.953994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:14.954008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:14.954071 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:14.955036 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.955579 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.956565 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.956755 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.956807 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:14.958989 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:14.958991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:14.958991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:14.958999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:14.960542 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:14.960976 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:14.960978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:14.960978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:14.961030 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:14.961377 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:14.961377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:14.961378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:14.961562 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:14.961622 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:14.961992 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 03:26:14.962103 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:26:14.962406 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:14.953141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41237 (local address 127.27.68.65:58009)
0929 03:26:14.953390 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:14.953421 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:14.953899 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:14.954128 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:14.954131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:14.954731 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:14.954938 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:14.954943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.955669 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.955669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:14.956474 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:14.956476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:14.956833 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:14.956834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:14.956943 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:14.957460 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:14.957470 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:14.958893 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:14.960616 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:14.960622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:14.960624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:14.960925 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:14.961101 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:14.961102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:14.961102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:14.961307 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:14.961622 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:14.961631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:14.961822 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 03:26:14.961953 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 03:26:14.962165 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260929 03:26:14.962899 14808 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41237
I20260929 03:26:14.963023 14808 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:14.963279 14808 heartbeater.cc:511] Master 127.27.68.126:41237 requested a full tablet report, sending...
I20260929 03:26:14.963853 14617 ts_manager.cc:194] Registered new tserver with Master: 4b737e492edf4a0789b4387ebc0a3483 (127.27.68.65:38339)
I20260929 03:26:14.964484 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.65@KRBTEST.COM'} at 127.27.68.65:58009
WARNING: no policy specified for kudu/127.27.68.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
Entry for principal kudu/127.27.68.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab.
I20260929 03:26:14.987732 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:15.070144 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:15.070362 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:15.070386 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:15.071574 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:15.071611 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:15.071640 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:15.073150 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.66.keytab
--principal=kudu/127.27.68.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:15.073364 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:15.073560 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:15.078703 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.078732 14813 server_base.cc:1034] running on GCE node
W20260929 03:26:15.078703 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.078732 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.079146 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:15.079429 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:15.080608 14813 hybrid_clock.cc:648] HybridClock initialized: now 1790652375080598 us; error 29 us; skew 500 ppm
Sep 29 03:26:15 dist-test-slave-fzwk krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652375, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:15.083168 14813 init.cc:377] Logged in from keytab as kudu/127.27.68.66@KRBTEST.COM (short username kudu)
I20260929 03:26:15.083658 14813 webserver.cc:460] Webserver started at http://127.27.68.66:46429/ using document root <none> and password file <none>
I20260929 03:26:15.083868 14813 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:15.083911 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:15.084004 14813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:15.085062 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "492f803c03264e499820ed65b7be3be4"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "b41a73173ca7a851fb771973965835af"
server_key_iv: "6f703d885c9b96ba2a8415e8bf3548a1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.085361 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "492f803c03264e499820ed65b7be3be4"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "b41a73173ca7a851fb771973965835af"
server_key_iv: "6f703d885c9b96ba2a8415e8bf3548a1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.086666 14813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:15.087425 14828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.087611 14813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:15.087723 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "492f803c03264e499820ed65b7be3be4"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "b41a73173ca7a851fb771973965835af"
server_key_iv: "6f703d885c9b96ba2a8415e8bf3548a1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.087805 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:15.111153 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:15.113682 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:15.113804 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:15.113986 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:15.114324 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:15.114383 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.114429 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:15.114452 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.120489 14813 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:38809
I20260929 03:26:15.120554 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:38809 every 8 connection(s)
I20260929 03:26:15.120942 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:26:15.123162 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14813
I20260929 03:26:15.123322 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:15.123538 27921 external_mini_cluster.cc:1428] Setting key 9e30593d168d827bd15d3359bc721f85
Sep 29 03:26:15 dist-test-slave-fzwk krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652375, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.66@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:15.131357 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.121804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:33687 (local address 127.27.68.126:41237)
0929 03:26:15.121878 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.121881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.122127 (+ 246us) server_negotiation.cc:407] Connection header received
0929 03:26:15.122621 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.122622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.122633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.122688 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.123942 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.124645 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.125780 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.125971 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.126028 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.128372 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.128374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.128374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:15.128382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.129855 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.130305 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.130306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.130307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.130368 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.130642 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.130643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.130643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.130763 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.130835 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.131136 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.131263 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.27.68.67@KRBTEST.COM; defaulting to no policy
I20260929 03:26:15.131556 14944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.121790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41237 (local address 127.27.68.66:33687)
0929 03:26:15.121987 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.122028 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.122530 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.122771 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.122775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.123530 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:15.123867 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.123882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.124746 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.124747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.125673 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.125676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.126063 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.126064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.126222 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.126817 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:15.126827 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:15.128268 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:15.129964 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:15.129973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:15.129975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:15.130217 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:15.130435 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:15.130436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:15.130437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:15.130564 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:15.130838 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:15.130846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:15.130999 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.131115 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.131307 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260929 03:26:15.132031 14942 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41237
I20260929 03:26:15.132150 14942 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:15.132354 14942 heartbeater.cc:511] Master 127.27.68.126:41237 requested a full tablet report, sending...
I20260929 03:26:15.132822 14617 ts_manager.cc:194] Registered new tserver with Master: 492f803c03264e499820ed65b7be3be4 (127.27.68.66:38809)
I20260929 03:26:15.133322 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.66@KRBTEST.COM'} at 127.27.68.66:33687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
Entry for principal kudu/127.27.68.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab.
I20260929 03:26:15.149863 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:15.233153 14947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:15.233368 14947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:15.233394 14947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:15.234738 14947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:15.234788 14947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:15.234817 14947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:15.236398 14947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.67.keytab
--principal=kudu/127.27.68.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:41237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:15.236666 14947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:15.236903 14947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:15.241866 14955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.241828 14953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.241840 14952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.242179 14947 server_base.cc:1034] running on GCE node
I20260929 03:26:15.242420 14947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:15.242702 14947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:15.243867 14947 hybrid_clock.cc:648] HybridClock initialized: now 1790652375243860 us; error 36 us; skew 500 ppm
Sep 29 03:26:15 dist-test-slave-fzwk krb5kdc[14572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652375, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:15.246382 14947 init.cc:377] Logged in from keytab as kudu/127.27.68.67@KRBTEST.COM (short username kudu)
I20260929 03:26:15.246876 14947 webserver.cc:460] Webserver started at http://127.27.68.67:39773/ using document root <none> and password file <none>
I20260929 03:26:15.247135 14947 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:15.247208 14947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:15.247400 14947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:15.248534 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "ae066c5063a1485786aaaac40cdb8e5f"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "0535884a5610a574b922b3e95d0a102c"
server_key_iv: "d4c9b49ced68210b9c5446dc116b68f1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.248909 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "ae066c5063a1485786aaaac40cdb8e5f"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "0535884a5610a574b922b3e95d0a102c"
server_key_iv: "d4c9b49ced68210b9c5446dc116b68f1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.250279 14947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:15.251027 14962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.251204 14947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:15.251338 14947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "ae066c5063a1485786aaaac40cdb8e5f"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "0535884a5610a574b922b3e95d0a102c"
server_key_iv: "d4c9b49ced68210b9c5446dc116b68f1"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.251430 14947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:15.271625 14947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:15.273774 14947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:15.273975 14947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:15.274233 14947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:15.274585 14947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:15.274642 14947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.274689 14947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:15.274714 14947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.281268 14947 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:40743
I20260929 03:26:15.281313 15075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:40743 every 8 connection(s)
I20260929 03:26:15.281677 14947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:26:15.285408 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 14947
I20260929 03:26:15.285488 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:15.285678 27921 external_mini_cluster.cc:1428] Setting key 2f1fa2607c3a8f5e930899c377203a06
Sep 29 03:26:15 dist-test-slave-fzwk krb5kdc[14572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652375, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.67@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:15.291182 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.282436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:34451 (local address 127.27.68.126:41237)
0929 03:26:15.282481 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.282487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.282689 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:26:15.283281 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.283283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.283296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.283351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.284367 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.284887 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.286077 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.286326 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.286422 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.288715 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.288716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.288717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:15.288723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.289925 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.290323 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.290324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.290324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.290375 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.290635 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.290635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.290635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.290759 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.290813 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.291052 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.291117 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:15.291502 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.282398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:41237 (local address 127.27.68.67:34451)
0929 03:26:15.282562 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.282600 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.283139 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.283432 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.283437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.284043 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:15.284249 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.284255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.284993 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.284994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.285957 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.285960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.286449 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.286450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.286601 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.287183 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:15.287192 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:15.288624 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:15.290019 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:15.290027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:15.290029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:15.290258 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:15.290432 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:15.290432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:15.290432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:15.290579 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:15.290816 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:15.290821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:15.290940 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.291022 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.291200 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260929 03:26:15.292220 15076 heartbeater.cc:347] Connected to a master server at 127.27.68.126:41237
I20260929 03:26:15.292336 15076 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:15.292572 15076 heartbeater.cc:511] Master 127.27.68.126:41237 requested a full tablet report, sending...
I20260929 03:26:15.293074 14617 ts_manager.cc:194] Registered new tserver with Master: ae066c5063a1485786aaaac40cdb8e5f (127.27.68.67:40743)
I20260929 03:26:15.293586 14617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.68.67@KRBTEST.COM'} at 127.27.68.67:34451
I20260929 03:26:15.297616 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:15.304543 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.298940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.27.68.126:41237)
0929 03:26:15.299030 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.299031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.299073 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:26:15.299130 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.299130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.299139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.299184 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.299784 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.300254 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.300947 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.301120 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.301167 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.302334 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.302334 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.302335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:15.302339 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.303568 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.303875 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.303876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.303876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.303923 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:15.304082 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:15.304083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:15.304083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:15.304219 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.304293 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.304393 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.304447 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:26:15.306257 14617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:15.307967 14617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:15.314769 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.27.68.65:38339)
0929 03:26:15.310848 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.310854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.310865 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:15.310910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.310912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.310929 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.311249 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:15.311328 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312511 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.313799 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314290 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314399 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314467 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.314629 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.314677 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260929 03:26:15.314762 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38339 (local address 127.0.0.1:35910)
0929 03:26:15.310499 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.310539 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.310606 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.311064 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.311067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.311069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:15.311146 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.311148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312625 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.312626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.313704 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.313706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314402 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.314495 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314509 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.314610 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.314670 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260929 03:26:15.314898 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.27.68.67:40743)
0929 03:26:15.310799 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.310805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.311088 (+ 283us) server_negotiation.cc:407] Connection header received
0929 03:26:15.311247 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.311250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.311268 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.311343 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:15.311521 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312228 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.313736 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314424 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314497 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314560 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.314773 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.314821 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260929 03:26:15.314939 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.27.68.66:38809)
0929 03:26:15.310992 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.310995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.311006 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:15.311058 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.311061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.311077 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.311152 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:15.311477 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312547 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.313673 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314424 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314497 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314550 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.314819 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.314858 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260929 03:26:15.315069 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:38809 (local address 127.0.0.1:38302)
0929 03:26:15.310882 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.310892 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.310939 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.311326 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.311327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.311330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:15.311393 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.311395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312656 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.312657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.313584 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.313587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314589 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.314663 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314670 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.314734 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.314779 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260929 03:26:15.315748 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.310426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:40743 (local address 127.0.0.1:55214)
0929 03:26:15.311015 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.311024 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.311116 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.311405 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.311406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.311408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:15.311441 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.311442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.312332 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.312333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.313638 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.313641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.314591 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.314592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.314663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.314673 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.315584 (+ 911us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.315657 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":39,"thread_start_us":75,"threads_started":1}
I20260929 03:26:15.315687 14742 tablet_service.cc:1467] Processing CreateTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 (DEFAULT_TABLE table=test-table [id=fed6e36d3ba0432299a0e230bfbcfa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:15.315784 14876 tablet_service.cc:1467] Processing CreateTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 (DEFAULT_TABLE table=test-table [id=fed6e36d3ba0432299a0e230bfbcfa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:15.316061 14876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d03c51bc81f4877940e997e9da2b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:15.316089 14742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d03c51bc81f4877940e997e9da2b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:15.316597 15010 tablet_service.cc:1467] Processing CreateTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 (DEFAULT_TABLE table=test-table [id=fed6e36d3ba0432299a0e230bfbcfa8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:15.316892 15010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d03c51bc81f4877940e997e9da2b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:15.318483 15093 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Bootstrap starting.
I20260929 03:26:15.319125 15093 tablet_bootstrap.cc:654] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:15.319636 15094 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: Bootstrap starting.
I20260929 03:26:15.319767 15093 log.cc:824] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:15.320104 15095 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Bootstrap starting.
I20260929 03:26:15.320297 15094 tablet_bootstrap.cc:654] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:15.320565 15093 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: No bootstrap required, opened a new log
I20260929 03:26:15.320655 15093 ts_tablet_manager.cc:1397] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:26:15.320758 15095 tablet_bootstrap.cc:654] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:15.320875 15094 log.cc:824] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:15.321362 15095 log.cc:824] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:15.321849 15094 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: No bootstrap required, opened a new log
I20260929 03:26:15.321919 15094 ts_tablet_manager.cc:1397] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:15.322062 15095 tablet_bootstrap.cc:492] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: No bootstrap required, opened a new log
I20260929 03:26:15.322187 15095 ts_tablet_manager.cc:1397] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:15.322808 15093 raft_consensus.cc:348] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.323007 15093 raft_consensus.cc:374] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:15.323045 15093 raft_consensus.cc:729] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492f803c03264e499820ed65b7be3be4, State: Initialized, Role: FOLLOWER
I20260929 03:26:15.323171 15093 consensus_queue.cc:260] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.323498 15093 ts_tablet_manager.cc:1428] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:26:15.323503 14942 heartbeater.cc:503] Master 127.27.68.126:41237 was elected leader, sending a full tablet report...
I20260929 03:26:15.323900 15095 raft_consensus.cc:348] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.324080 15095 raft_consensus.cc:374] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:15.324116 15095 raft_consensus.cc:729] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae066c5063a1485786aaaac40cdb8e5f, State: Initialized, Role: FOLLOWER
I20260929 03:26:15.324039 15094 raft_consensus.cc:348] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.324155 15094 raft_consensus.cc:374] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:15.324177 15094 raft_consensus.cc:729] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b737e492edf4a0789b4387ebc0a3483, State: Initialized, Role: FOLLOWER
I20260929 03:26:15.324221 15095 consensus_queue.cc:260] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.324308 15094 consensus_queue.cc:260] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.324563 15076 heartbeater.cc:503] Master 127.27.68.126:41237 was elected leader, sending a full tablet report...
I20260929 03:26:15.324584 15094 ts_tablet_manager.cc:1428] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:26:15.324718 14808 heartbeater.cc:503] Master 127.27.68.126:41237 was elected leader, sending a full tablet report...
I20260929 03:26:15.324756 15095 ts_tablet_manager.cc:1428] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 03:26:15.371690 14943 tablet.cc:2369] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:26:15.409883 15099 raft_consensus.cc:482] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:15.410017 15099 raft_consensus.cc:504] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.410481 15099 leader_election.cc:288] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b737e492edf4a0789b4387ebc0a3483 (127.27.68.65:38339), ae066c5063a1485786aaaac40cdb8e5f (127.27.68.67:40743)
I20260929 03:26:15.413273 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.410717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:37305 (local address 127.27.68.65:38339)
0929 03:26:15.410770 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.410772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.410803 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:26:15.410837 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.410838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.410846 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.410907 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:15.411075 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.411609 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.412534 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.412914 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.413001 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.413011 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.413169 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.413208 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:26:15.413273 14944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.410650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38339 (local address 127.27.68.66:37305)
0929 03:26:15.410707 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.410719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.410763 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.410912 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.410913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.410914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:15.410993 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.410995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.411692 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.411693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.412435 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.412437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.412999 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.413001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.413085 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.413088 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.413151 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.413195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:26:15.413403 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.410765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:43091 (local address 127.27.68.67:40743)
0929 03:26:15.410810 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.410812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.410930 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:26:15.410997 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.410998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.411009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.411063 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:15.411260 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.411725 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.412661 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.413061 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.413144 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.413152 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.413302 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.413340 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:26:15.413416 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.410650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:40743 (local address 127.27.68.66:43091)
0929 03:26:15.410842 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.410851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.410938 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.411077 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.411077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.411078 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:15.411155 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.411157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.411810 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.411811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.412567 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.412569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.413145 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.413146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.413225 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.413228 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.413295 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.413335 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":10,"thread_start_us":113,"threads_started":1}
I20260929 03:26:15.413688 14762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d03c51bc81f4877940e997e9da2b1b5" candidate_uuid: "492f803c03264e499820ed65b7be3be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b737e492edf4a0789b4387ebc0a3483" is_pre_election: true
I20260929 03:26:15.413820 14762 raft_consensus.cc:2454] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492f803c03264e499820ed65b7be3be4 in term 0.
I20260929 03:26:15.414057 14831 leader_election.cc:302] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492f803c03264e499820ed65b7be3be4, 4b737e492edf4a0789b4387ebc0a3483; no voters:
I20260929 03:26:15.414083 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d03c51bc81f4877940e997e9da2b1b5" candidate_uuid: "492f803c03264e499820ed65b7be3be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae066c5063a1485786aaaac40cdb8e5f" is_pre_election: true
I20260929 03:26:15.414214 15030 raft_consensus.cc:2454] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492f803c03264e499820ed65b7be3be4 in term 0.
I20260929 03:26:15.414211 15099 raft_consensus.cc:2781] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:15.414271 15099 raft_consensus.cc:482] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:15.414302 15099 raft_consensus.cc:3037] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:15.415027 15099 raft_consensus.cc:504] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.415190 15099 leader_election.cc:288] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [CANDIDATE]: Term 1 election: Requested vote from peers 4b737e492edf4a0789b4387ebc0a3483 (127.27.68.65:38339), ae066c5063a1485786aaaac40cdb8e5f (127.27.68.67:40743)
I20260929 03:26:15.415393 15030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d03c51bc81f4877940e997e9da2b1b5" candidate_uuid: "492f803c03264e499820ed65b7be3be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae066c5063a1485786aaaac40cdb8e5f"
I20260929 03:26:15.415409 14762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d03c51bc81f4877940e997e9da2b1b5" candidate_uuid: "492f803c03264e499820ed65b7be3be4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b737e492edf4a0789b4387ebc0a3483"
I20260929 03:26:15.415495 15030 raft_consensus.cc:3037] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:15.415490 14762 raft_consensus.cc:3037] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:15.416209 15030 raft_consensus.cc:2454] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492f803c03264e499820ed65b7be3be4 in term 1.
I20260929 03:26:15.416257 14762 raft_consensus.cc:2454] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492f803c03264e499820ed65b7be3be4 in term 1.
I20260929 03:26:15.416431 14832 leader_election.cc:302] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 492f803c03264e499820ed65b7be3be4, ae066c5063a1485786aaaac40cdb8e5f; no voters:
I20260929 03:26:15.416575 15099 raft_consensus.cc:2781] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:15.416755 15099 raft_consensus.cc:686] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 1 LEADER]: Becoming Leader. State: Replica: 492f803c03264e499820ed65b7be3be4, State: Running, Role: LEADER
I20260929 03:26:15.416862 15099 consensus_queue.cc:237] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } }
I20260929 03:26:15.417749 14617 catalog_manager.cc:5495] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 492f803c03264e499820ed65b7be3be4 (127.27.68.66). New cstate: current_term: 1 leader_uuid: "492f803c03264e499820ed65b7be3be4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492f803c03264e499820ed65b7be3be4" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 38809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 } health_report { overall_health: UNKNOWN } } }
I20260929 03:26:15.434944 15092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.432280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.27.68.66:38809)
0929 03:26:15.432332 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.432334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.432344 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:15.432396 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.432396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.432416 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.432497 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:15.432680 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.433288 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.434094 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.434308 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.434385 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.434573 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:15.434688 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:15.434788 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.434807 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.434831 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:15.436851 15030 raft_consensus.cc:1261] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 1 FOLLOWER]: Refusing update from remote peer 492f803c03264e499820ed65b7be3be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:15.436860 14762 raft_consensus.cc:1261] T 5d03c51bc81f4877940e997e9da2b1b5 P 4b737e492edf4a0789b4387ebc0a3483 [term 1 FOLLOWER]: Refusing update from remote peer 492f803c03264e499820ed65b7be3be4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:15.437258 15103 consensus_queue.cc:1035] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b737e492edf4a0789b4387ebc0a3483" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 38339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:15.437408 15099 consensus_queue.cc:1035] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae066c5063a1485786aaaac40cdb8e5f" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 40743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:15.439936 15107 mvcc.cc:205] Tried to move back new op lower bound from 7334512129786540032 to 7334512129708122112. Current Snapshot: MvccSnapshot[applied={T|T < 7334512129786540032}]
I20260929 03:26:15.441690 15108 mvcc.cc:205] Tried to move back new op lower bound from 7334512129786540032 to 7334512129708122112. Current Snapshot: MvccSnapshot[applied={T|T < 7334512129786540032}]
I20260929 03:26:15.441924 14617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:15.442014 14617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:15.442727 14617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4866504264e49a89e6cb7e3c6e883d6: Sending DeleteTablet for 3 replicas of tablet 5d03c51bc81f4877940e997e9da2b1b5
I20260929 03:26:15.443092 14742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:15 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:35910
I20260929 03:26:15.443163 15010 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:15 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:55214
I20260929 03:26:15.443153 14876 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d03c51bc81f4877940e997e9da2b1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:15 UTC) from {username='kudu', principal='kudu/127.27.68.126@KRBTEST.COM'} at 127.0.0.1:38302
I20260929 03:26:15.443456 15118 tablet_replica.cc:330] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: stopping tablet replica
I20260929 03:26:15.443574 15118 raft_consensus.cc:2229] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:26:15.443603 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14679
I20260929 03:26:15.443670 15118 raft_consensus.cc:2258] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:15.446072 15118 ts_tablet_manager.cc:1905] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:15.447885 15118 ts_tablet_manager.cc:1918] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:15.447970 15118 log.cc:1196] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/5d03c51bc81f4877940e997e9da2b1b5
I20260929 03:26:15.448258 15118 ts_tablet_manager.cc:1939] T 5d03c51bc81f4877940e997e9da2b1b5 P 492f803c03264e499820ed65b7be3be4: Deleting consensus metadata
I20260929 03:26:15.448791 14602 catalog_manager.cc:4841] TS 492f803c03264e499820ed65b7be3be4 (127.27.68.66:38809): tablet 5d03c51bc81f4877940e997e9da2b1b5 (table test-table [id=fed6e36d3ba0432299a0e230bfbcfa8e]) successfully deleted
W20260929 03:26:15.450490 14603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:38339 (error 108)
I20260929 03:26:15.451089 15116 tablet_replica.cc:330] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: stopping tablet replica
I20260929 03:26:15.451095 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.450941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:38339 (local address 127.0.0.1:35920)
0929 03:26:15.451009 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.451058 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 03:26:15.451268 15116 raft_consensus.cc:2229] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 03:26:15.451366 14603 catalog_manager.cc:4584] TS 4b737e492edf4a0789b4387ebc0a3483 (127.27.68.65:38339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d03c51bc81f4877940e997e9da2b1b5: Network error: Client connection negotiation failed: client connection to 127.27.68.65:38339: connect: Connection refused (error 111)
I20260929 03:26:15.451404 15116 raft_consensus.cc:2258] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:15.451635 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14813
I20260929 03:26:15.451922 15116 ts_tablet_manager.cc:1905] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:15.454051 15116 ts_tablet_manager.cc:1918] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:15.454140 15116 log.cc:1196] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/5d03c51bc81f4877940e997e9da2b1b5
I20260929 03:26:15.454458 15116 ts_tablet_manager.cc:1939] T 5d03c51bc81f4877940e997e9da2b1b5 P ae066c5063a1485786aaaac40cdb8e5f: Deleting consensus metadata
I20260929 03:26:15.454924 14604 catalog_manager.cc:4841] TS ae066c5063a1485786aaaac40cdb8e5f (127.27.68.67:40743): tablet 5d03c51bc81f4877940e997e9da2b1b5 (table test-table [id=fed6e36d3ba0432299a0e230bfbcfa8e]) successfully deleted
I20260929 03:26:15.459152 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14947
I20260929 03:26:15.464073 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 14585
2026-09-29T03:26:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1996 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:15Z Disabled control of system clock
I20260929 03:26:15.491915 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:45041
--webserver_interface=127.27.68.126
--webserver_port=0
--builtin_ntp_servers=127.27.68.84:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:15.573503 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:15.573753 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:15.573778 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:15.575052 15123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:15.575109 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:15.575121 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:15.575134 15123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:15.575187 15123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:15.576782 15123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:45041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:15.577092 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:15.577425 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:15.582547 15128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.582731 15123 server_base.cc:1034] running on GCE node
W20260929 03:26:15.582572 15131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.582568 15129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.583132 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:15.583467 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:15.584619 15123 hybrid_clock.cc:648] HybridClock initialized: now 1790652375584602 us; error 34 us; skew 500 ppm
I20260929 03:26:15.585994 15123 webserver.cc:460] Webserver started at http://127.27.68.126:34513/ using document root <none> and password file <none>
I20260929 03:26:15.586272 15123 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:15.586346 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:15.586498 15123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:15.587757 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "181fa97a8e0945d985458fe558cf72e3"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8ce645d91bbddc4b65dc06a3b13cf98f"
server_key_iv: "e2567a5e53e4bf3e24990aae89dafd66"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.588128 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "181fa97a8e0945d985458fe558cf72e3"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8ce645d91bbddc4b65dc06a3b13cf98f"
server_key_iv: "e2567a5e53e4bf3e24990aae89dafd66"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.589497 15123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:15.590211 15137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.590379 15123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:15.590485 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "181fa97a8e0945d985458fe558cf72e3"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8ce645d91bbddc4b65dc06a3b13cf98f"
server_key_iv: "e2567a5e53e4bf3e24990aae89dafd66"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.590566 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:15.606567 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:15.606923 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:15.607093 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:15.611135 15123 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:45041
I20260929 03:26:15.611191 15189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:45041 every 8 connection(s)
I20260929 03:26:15.611613 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:15.612609 15190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:15.614907 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: Bootstrap starting.
I20260929 03:26:15.615548 15190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:15.615861 15190 log.cc:824] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:15.616566 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: No bootstrap required, opened a new log
I20260929 03:26:15.616582 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15123
I20260929 03:26:15.616710 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:15.616889 27921 external_mini_cluster.cc:1428] Setting key a6cc6ff33197f6614ff62c899b16d3a5
I20260929 03:26:15.618419 15190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } }
I20260929 03:26:15.618588 15190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:15.618634 15190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181fa97a8e0945d985458fe558cf72e3, State: Initialized, Role: FOLLOWER
I20260929 03:26:15.618767 15190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } }
I20260929 03:26:15.618854 15190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:15.618896 15190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:15.618952 15190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:15.619592 15190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } }
I20260929 03:26:15.619712 15190 leader_election.cc:302] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181fa97a8e0945d985458fe558cf72e3; no voters:
I20260929 03:26:15.619940 15190 leader_election.cc:288] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:15.619979 15195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:15.620083 15195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [term 1 LEADER]: Becoming Leader. State: Replica: 181fa97a8e0945d985458fe558cf72e3, State: Running, Role: LEADER
I20260929 03:26:15.620168 15195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } }
I20260929 03:26:15.620219 15190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:15.620508 15195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181fa97a8e0945d985458fe558cf72e3. Latest consensus state: current_term: 1 leader_uuid: "181fa97a8e0945d985458fe558cf72e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } } }
I20260929 03:26:15.620523 15196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181fa97a8e0945d985458fe558cf72e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181fa97a8e0945d985458fe558cf72e3" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45041 } } }
I20260929 03:26:15.620600 15195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:15.620631 15196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3 [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:15.622011 15211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:15.622072 15211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:15.622457 15200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:15.622673 15200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:15.623829 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.617478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.27.68.126:45041)
0929 03:26:15.617690 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.617700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.617762 (+ 62us) server_negotiation.cc:407] Connection header received
0929 03:26:15.618407 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.618413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.618476 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.618616 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.618821 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.620398 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.621587 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.622125 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.622244 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.623029 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.623034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.623034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:15.623285 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.623336 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:15.623352 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.623421 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.623490 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.623587 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260929 03:26:15.624485 15200 catalog_manager.cc:1347] Generated new cluster ID: 7e42fb1ccf814a4482161c7edb257827
I20260929 03:26:15.624547 15200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:15.679126 15200 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:15.679904 15200 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:15.690631 15200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: Generated new TSK 0
I20260929 03:26:15.690883 15200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:15.693529 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.65:0
--local_ip_for_outbound_sockets=127.27.68.65
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--builtin_ntp_servers=127.27.68.84:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:15.776427 15214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:15.776652 15214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:15.776675 15214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:15.777905 15214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:15.777954 15214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:15.777982 15214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.65
I20260929 03:26:15.779456 15214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.68.65
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.65
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:15.779738 15214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:15.780072 15214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:15.785281 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.785339 15220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.785285 15219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.785645 15214 server_base.cc:1034] running on GCE node
I20260929 03:26:15.785918 15214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:15.786185 15214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:15.787338 15214 hybrid_clock.cc:648] HybridClock initialized: now 1790652375787327 us; error 31 us; skew 500 ppm
I20260929 03:26:15.788614 15214 webserver.cc:460] Webserver started at http://127.27.68.65:39065/ using document root <none> and password file <none>
I20260929 03:26:15.788831 15214 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:15.788874 15214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:15.788970 15214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:15.790119 15214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data/instance:
uuid: "1701253c4ef24402bdcd7ab406b2d90e"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8c26dfcc99fcd84113d3c58803614e38"
server_key_iv: "d2e4674afb5b49c3df29a60498494392"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.790431 15214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance:
uuid: "1701253c4ef24402bdcd7ab406b2d90e"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8c26dfcc99fcd84113d3c58803614e38"
server_key_iv: "d2e4674afb5b49c3df29a60498494392"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.791779 15214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:15.792476 15228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.792668 15214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:15.792770 15214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal
uuid: "1701253c4ef24402bdcd7ab406b2d90e"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "8c26dfcc99fcd84113d3c58803614e38"
server_key_iv: "d2e4674afb5b49c3df29a60498494392"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.792860 15214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:15.804383 15214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:15.804726 15214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:15.804879 15214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:15.805116 15214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:15.805437 15214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:15.805492 15214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.805536 15214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:15.805574 15214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.811403 15214 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.65:40681
I20260929 03:26:15.811480 15341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.65:40681 every 8 connection(s)
I20260929 03:26:15.811854 15214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/data/info.pb
I20260929 03:26:15.817433 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.812694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:42785 (local address 127.27.68.126:45041)
0929 03:26:15.812759 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.812762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.812996 (+ 234us) server_negotiation.cc:407] Connection header received
0929 03:26:15.813549 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.813550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.813566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.813620 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.814341 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.814860 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.815917 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.816123 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.816199 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.817114 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.817116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.817116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:15.817125 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.817141 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:15.817145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.817218 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.817318 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.817352 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:26:15.817651 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.812702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:45041 (local address 127.27.68.65:42785)
0929 03:26:15.812869 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.812903 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.813429 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.813712 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.813716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.813725 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:15.814240 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.814245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.814973 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.814974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.815808 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.815811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.816220 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.816221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.816359 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.816942 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:15.816942 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:15.816973 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:15.816997 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:15.817216 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:15.817223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:15.817227 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.817319 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.817429 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260929 03:26:15.818166 15342 heartbeater.cc:347] Connected to a master server at 127.27.68.126:45041
I20260929 03:26:15.818262 15342 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:15.818418 15342 heartbeater.cc:511] Master 127.27.68.126:45041 requested a full tablet report, sending...
I20260929 03:26:15.818828 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15214
I20260929 03:26:15.818936 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-0/wal/instance
I20260929 03:26:15.819126 27921 external_mini_cluster.cc:1428] Setting key a60cf5e6b3d6f26b39f9efa2294b6412
I20260929 03:26:15.819567 15154 ts_manager.cc:194] Registered new tserver with Master: 1701253c4ef24402bdcd7ab406b2d90e (127.27.68.65:40681)
I20260929 03:26:15.820241 15154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.65:42785
I20260929 03:26:15.820268 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.66:0
--local_ip_for_outbound_sockets=127.27.68.66
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--builtin_ntp_servers=127.27.68.84:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:15.902561 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:15.902822 15345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:15.902858 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:15.904115 15345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:15.904188 15345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:15.904232 15345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.66
I20260929 03:26:15.905649 15345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.68.66
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.66
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:15.905941 15345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:15.906270 15345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:15.911265 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.911290 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:15.911265 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:15.911501 15345 server_base.cc:1034] running on GCE node
I20260929 03:26:15.911769 15345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:15.912014 15345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:15.913192 15345 hybrid_clock.cc:648] HybridClock initialized: now 1790652375913177 us; error 47 us; skew 500 ppm
I20260929 03:26:15.914424 15345 webserver.cc:460] Webserver started at http://127.27.68.66:39779/ using document root <none> and password file <none>
I20260929 03:26:15.914695 15345 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:15.914767 15345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:15.914923 15345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:15.916113 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data/instance:
uuid: "bf61d90814ef436495b5464cdc8dcdd7"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "7c2189682f6900607a9e46c2fdc3b937"
server_key_iv: "977e25d548fa75c7ea972e75aa540090"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.916481 15345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance:
uuid: "bf61d90814ef436495b5464cdc8dcdd7"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "7c2189682f6900607a9e46c2fdc3b937"
server_key_iv: "977e25d548fa75c7ea972e75aa540090"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.917822 15345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:26:15.918547 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.918727 15345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:15.918828 15345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal
uuid: "bf61d90814ef436495b5464cdc8dcdd7"
format_stamp: "Formatted at 2026-09-29 03:26:15 on dist-test-slave-fzwk"
server_key: "7c2189682f6900607a9e46c2fdc3b937"
server_key_iv: "977e25d548fa75c7ea972e75aa540090"
server_key_version: "encryptionkey@0"
I20260929 03:26:15.918913 15345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:15.928489 15345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:15.928797 15345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:15.928951 15345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:15.929173 15345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:15.929504 15345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:15.929565 15345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.929608 15345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:15.929636 15345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:15.935523 15345 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.66:36099
I20260929 03:26:15.935578 15472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.66:36099 every 8 connection(s)
I20260929 03:26:15.935906 15345 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/data/info.pb
I20260929 03:26:15.940763 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.936630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.66:46771 (local address 127.27.68.126:45041)
0929 03:26:15.936673 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:15.936675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:15.936910 (+ 235us) server_negotiation.cc:407] Connection header received
0929 03:26:15.937378 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:15.937379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:15.937388 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:15.937442 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:15.938090 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.938583 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.939562 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.939743 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.939796 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.940528 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:15.940529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:15.940529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:15.940534 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:15.940554 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:15.940558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:15.940599 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:15.940680 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:26:15.940704 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:26:15.940949 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:15.936600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:45041 (local address 127.27.68.66:46771)
0929 03:26:15.936783 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:15.936824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:15.937297 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:15.937503 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:15.937506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:15.937514 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:15.938006 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.938009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.938663 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.938664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.939475 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:15.939477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:15.939817 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:15.939818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:15.939923 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:15.940402 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:15.940402 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:15.940431 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:15.940453 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:15.940611 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:15.940615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:15.940617 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:26:15.940669 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:26:15.940747 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260929 03:26:15.941504 15473 heartbeater.cc:347] Connected to a master server at 127.27.68.126:45041
I20260929 03:26:15.941613 15473 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:15.941797 15473 heartbeater.cc:511] Master 127.27.68.126:45041 requested a full tablet report, sending...
I20260929 03:26:15.942250 15154 ts_manager.cc:194] Registered new tserver with Master: bf61d90814ef436495b5464cdc8dcdd7 (127.27.68.66:36099)
I20260929 03:26:15.942596 15154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.66:46771
I20260929 03:26:15.945571 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15345
I20260929 03:26:15.945657 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal/instance
I20260929 03:26:15.945802 27921 external_mini_cluster.cc:1428] Setting key 560ba34205432a4a50b46ce8d7e9931d
I20260929 03:26:15.946678 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.68.67:0
--local_ip_for_outbound_sockets=127.27.68.67
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--builtin_ntp_servers=127.27.68.84:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:26:16.029279 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:16.029516 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:16.029541 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:16.030694 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:16.030732 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:16.030758 15476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.68.67
I20260929 03:26:16.032248 15476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.68.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.68.67
--webserver_port=0
--tserver_master_addrs=127.27.68.126:45041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.68.67
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:16.032485 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:16.032768 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:16.037837 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:16.037827 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:16.037827 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:16.038154 15476 server_base.cc:1034] running on GCE node
I20260929 03:26:16.038393 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:16.038628 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:16.039800 15476 hybrid_clock.cc:648] HybridClock initialized: now 1790652376039792 us; error 36 us; skew 500 ppm
I20260929 03:26:16.041256 15476 webserver.cc:460] Webserver started at http://127.27.68.67:33531/ using document root <none> and password file <none>
I20260929 03:26:16.041548 15476 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:16.041626 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:16.041783 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:16.042932 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data/instance:
uuid: "b4e517794331410e873c93ca307f838d"
format_stamp: "Formatted at 2026-09-29 03:26:16 on dist-test-slave-fzwk"
server_key: "f234006cbb555d4c271e69e5ffa8b598"
server_key_iv: "5911e882bbd839c7dded093cdaf571be"
server_key_version: "encryptionkey@0"
I20260929 03:26:16.043378 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance:
uuid: "b4e517794331410e873c93ca307f838d"
format_stamp: "Formatted at 2026-09-29 03:26:16 on dist-test-slave-fzwk"
server_key: "f234006cbb555d4c271e69e5ffa8b598"
server_key_iv: "5911e882bbd839c7dded093cdaf571be"
server_key_version: "encryptionkey@0"
I20260929 03:26:16.044771 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:16.045569 15490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:16.045753 15476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:16.045862 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal
uuid: "b4e517794331410e873c93ca307f838d"
format_stamp: "Formatted at 2026-09-29 03:26:16 on dist-test-slave-fzwk"
server_key: "f234006cbb555d4c271e69e5ffa8b598"
server_key_iv: "5911e882bbd839c7dded093cdaf571be"
server_key_version: "encryptionkey@0"
I20260929 03:26:16.045946 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:16.058354 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:16.058710 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:16.058883 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:16.059127 15476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:26:16.059518 15476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:26:16.059576 15476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:16.059621 15476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:26:16.059645 15476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:16.065580 15476 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.67:43769
I20260929 03:26:16.065625 15603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.67:43769 every 8 connection(s)
I20260929 03:26:16.066051 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/data/info.pb
I20260929 03:26:16.071583 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15476
I20260929 03:26:16.071719 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal/instance
I20260929 03:26:16.071710 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.066916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.67:45783 (local address 127.27.68.126:45041)
0929 03:26:16.066984 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.066986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.067234 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:26:16.067840 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.067841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.067856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.067909 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:16.068643 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.069175 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.070199 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.070419 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.070496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.071380 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:16.071381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:16.071382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:16.071387 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:16.071405 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:16.071410 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:16.071465 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.071587 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.071625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:26:16.071944 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.066906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:45041 (local address 127.27.68.67:45783)
0929 03:26:16.067092 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.067127 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.067693 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.068000 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.068005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.068022 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:16.068532 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.068536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.069281 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.069282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.070096 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.070098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.070519 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.070519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.070669 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.071208 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:26:16.071209 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:16.071258 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:26:16.071282 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:16.071472 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:16.071477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:16.071482 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.071583 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.071698 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260929 03:26:16.071939 27921 external_mini_cluster.cc:1428] Setting key d81e2a46917f77660d3443cfd5829fb2
I20260929 03:26:16.072571 15604 heartbeater.cc:347] Connected to a master server at 127.27.68.126:45041
I20260929 03:26:16.072696 15604 heartbeater.cc:464] Registering TS with master...
I20260929 03:26:16.072908 15604 heartbeater.cc:511] Master 127.27.68.126:45041 requested a full tablet report, sending...
I20260929 03:26:16.073280 15154 ts_manager.cc:194] Registered new tserver with Master: b4e517794331410e873c93ca307f838d (127.27.68.67:43769)
I20260929 03:26:16.073626 15154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.68.67:45783
I20260929 03:26:16.084157 27921 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:26:16.088743 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.085623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.27.68.126:45041)
0929 03:26:16.085716 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.085718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.085812 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:26:16.085889 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.085890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.085903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.085957 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:16.086157 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.086666 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.087500 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.087709 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.087791 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.088471 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:16.088472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:16.088473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:16.088478 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:16.088493 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:16.088498 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:16.088563 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.088618 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.088646 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 03:26:16.090504 15154 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:26:16.092268 15154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:26:16.099239 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.27.68.65:40681)
0929 03:26:16.094980 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.094988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.095143 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:26:16.095186 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.095189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.095266 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.095338 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:16.095601 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096847 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098249 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.098753 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098858 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.098964 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.099078 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.099116 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260929 03:26:16.099202 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40681 (local address 127.0.0.1:38574)
0929 03:26:16.094906 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.094965 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.095047 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.095435 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.095440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.095443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:16.095517 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.095520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096930 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.096931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.098160 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.098162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.098877 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.098963 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.098979 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.099062 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.099118 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":28,"thread_start_us":62,"threads_started":1}
I20260929 03:26:16.099397 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.27.68.67:43769)
0929 03:26:16.095133 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.095137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.095147 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:16.095191 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.095193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.095205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.095507 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:16.095749 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096600 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098107 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.098809 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098882 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.098967 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.099268 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.099310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260929 03:26:16.099397 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.27.68.66:36099)
0929 03:26:16.095347 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.095352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.095362 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:16.095401 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.095403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.095419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.095501 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:16.095731 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096843 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098044 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.098805 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.098885 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.098978 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.099268 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.099310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":86,"threads_started":1}
I20260929 03:26:16.099488 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:36099 (local address 127.0.0.1:50512)
0929 03:26:16.094964 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.094975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.095055 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.095571 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.095572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.095574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:16.095653 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.095654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096945 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.096946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.097927 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.097931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.099006 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.099008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.099091 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.099098 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.099150 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.099200 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":70,"thread_start_us":74,"threads_started":1}
I20260929 03:26:16.099584 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.094798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43769 (local address 127.0.0.1:45130)
0929 03:26:16.095034 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.095043 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.095083 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.095592 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.095593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.095595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:16.095662 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.095664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.096696 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.096697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.098030 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.098033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.099008 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.099009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.099091 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.099099 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.099150 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.099198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":55,"thread_start_us":83,"threads_started":1}
I20260929 03:26:16.100276 15407 tablet_service.cc:1467] Processing CreateTablet for tablet d45fec9e302e41578a200ac310f0a3e5 (DEFAULT_TABLE table=test-table [id=c25e83f8eb60420c9e621ecc2dced846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:16.100430 15276 tablet_service.cc:1467] Processing CreateTablet for tablet d45fec9e302e41578a200ac310f0a3e5 (DEFAULT_TABLE table=test-table [id=c25e83f8eb60420c9e621ecc2dced846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:16.100495 15538 tablet_service.cc:1467] Processing CreateTablet for tablet d45fec9e302e41578a200ac310f0a3e5 (DEFAULT_TABLE table=test-table [id=c25e83f8eb60420c9e621ecc2dced846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:26:16.100690 15407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45fec9e302e41578a200ac310f0a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:16.100781 15538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45fec9e302e41578a200ac310f0a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:16.100816 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45fec9e302e41578a200ac310f0a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:16.103724 15618 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: Bootstrap starting.
I20260929 03:26:16.104169 15619 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Bootstrap starting.
I20260929 03:26:16.104573 15618 tablet_bootstrap.cc:654] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:16.104831 15619 tablet_bootstrap.cc:654] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:16.104907 15618 log.cc:824] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:16.105278 15619 log.cc:824] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:16.105661 15620 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Bootstrap starting.
I20260929 03:26:16.105904 15618 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: No bootstrap required, opened a new log
I20260929 03:26:16.105991 15618 ts_tablet_manager.cc:1397] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:26:16.106339 15620 tablet_bootstrap.cc:654] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:16.106660 15620 log.cc:824] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:16.107510 15620 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: No bootstrap required, opened a new log
I20260929 03:26:16.107599 15620 ts_tablet_manager.cc:1397] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:26:16.107837 15619 tablet_bootstrap.cc:492] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: No bootstrap required, opened a new log
I20260929 03:26:16.107913 15619 ts_tablet_manager.cc:1397] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 03:26:16.108675 15618 raft_consensus.cc:348] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.108852 15618 raft_consensus.cc:374] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:16.108898 15618 raft_consensus.cc:729] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1701253c4ef24402bdcd7ab406b2d90e, State: Initialized, Role: FOLLOWER
I20260929 03:26:16.109015 15618 consensus_queue.cc:260] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.109315 15618 ts_tablet_manager.cc:1428] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:26:16.109338 15342 heartbeater.cc:503] Master 127.27.68.126:45041 was elected leader, sending a full tablet report...
I20260929 03:26:16.110383 15620 raft_consensus.cc:348] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.110566 15620 raft_consensus.cc:374] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:16.110500 15619 raft_consensus.cc:348] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.110615 15620 raft_consensus.cc:729] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf61d90814ef436495b5464cdc8dcdd7, State: Initialized, Role: FOLLOWER
I20260929 03:26:16.110635 15619 raft_consensus.cc:374] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:16.110669 15619 raft_consensus.cc:729] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e517794331410e873c93ca307f838d, State: Initialized, Role: FOLLOWER
I20260929 03:26:16.110743 15620 consensus_queue.cc:260] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.110771 15619 consensus_queue.cc:260] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.111028 15620 ts_tablet_manager.cc:1428] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:26:16.111028 15619 ts_tablet_manager.cc:1428] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:26:16.111075 15604 heartbeater.cc:503] Master 127.27.68.126:45041 was elected leader, sending a full tablet report...
I20260929 03:26:16.111284 15473 heartbeater.cc:503] Master 127.27.68.126:45041 was elected leader, sending a full tablet report...
I20260929 03:26:16.126368 15624 raft_consensus.cc:482] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:26:16.126489 15624 raft_consensus.cc:504] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.126796 15624 leader_election.cc:288] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf61d90814ef436495b5464cdc8dcdd7 (127.27.68.66:36099), b4e517794331410e873c93ca307f838d (127.27.68.67:43769)
I20260929 03:26:16.130756 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.127255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:34473 (local address 127.27.68.67:43769)
0929 03:26:16.127300 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.127302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.127375 (+ 73us) server_negotiation.cc:407] Connection header received
0929 03:26:16.127427 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.127427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.127434 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.127504 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:16.127639 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.128321 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.129816 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.130372 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.130455 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.130463 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.130637 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.130675 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:26:16.130756 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.127089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.68.65:50759 (local address 127.27.68.66:36099)
0929 03:26:16.127152 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.127154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.127164 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:16.127195 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.127196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.127206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.127282 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:26:16.127488 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.128263 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.129816 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.130372 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.130455 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.130463 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.130637 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.130675 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:26:16.130781 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.126985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.66:36099 (local address 127.27.68.65:50759)
0929 03:26:16.127043 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.127057 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.127113 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.127313 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.127314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.127316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:16.127413 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.127415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.128390 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.128391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.129716 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.129720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.130468 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.130470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.130550 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.130555 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.130617 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.130684 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":59}
I20260929 03:26:16.130787 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.127123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.67:43769 (local address 127.27.68.65:34473)
0929 03:26:16.127299 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.127309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:16.127370 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:16.127518 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:16.127519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:16.127520 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:26:16.127576 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.127577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.128394 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.128395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.129723 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:16.129725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.130484 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.130485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:16.130550 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.130555 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:26:16.130617 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:26:16.130677 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":92,"thread_start_us":127,"threads_started":1}
I20260929 03:26:16.131168 15427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fec9e302e41578a200ac310f0a3e5" candidate_uuid: "1701253c4ef24402bdcd7ab406b2d90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf61d90814ef436495b5464cdc8dcdd7" is_pre_election: true
I20260929 03:26:16.131387 15427 raft_consensus.cc:2454] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1701253c4ef24402bdcd7ab406b2d90e in term 0.
I20260929 03:26:16.131341 15558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fec9e302e41578a200ac310f0a3e5" candidate_uuid: "1701253c4ef24402bdcd7ab406b2d90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e517794331410e873c93ca307f838d" is_pre_election: true
I20260929 03:26:16.131506 15558 raft_consensus.cc:2454] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1701253c4ef24402bdcd7ab406b2d90e in term 0.
I20260929 03:26:16.131620 15232 leader_election.cc:302] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1701253c4ef24402bdcd7ab406b2d90e, bf61d90814ef436495b5464cdc8dcdd7; no voters:
I20260929 03:26:16.131762 15624 raft_consensus.cc:2781] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:26:16.131852 15624 raft_consensus.cc:482] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:26:16.131882 15624 raft_consensus.cc:3037] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:16.132647 15624 raft_consensus.cc:504] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.132786 15624 leader_election.cc:288] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [CANDIDATE]: Term 1 election: Requested vote from peers bf61d90814ef436495b5464cdc8dcdd7 (127.27.68.66:36099), b4e517794331410e873c93ca307f838d (127.27.68.67:43769)
I20260929 03:26:16.132948 15427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fec9e302e41578a200ac310f0a3e5" candidate_uuid: "1701253c4ef24402bdcd7ab406b2d90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf61d90814ef436495b5464cdc8dcdd7"
I20260929 03:26:16.132979 15558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45fec9e302e41578a200ac310f0a3e5" candidate_uuid: "1701253c4ef24402bdcd7ab406b2d90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e517794331410e873c93ca307f838d"
I20260929 03:26:16.133039 15558 raft_consensus.cc:3037] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:16.133055 15427 raft_consensus.cc:3037] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:16.133811 15427 raft_consensus.cc:2454] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1701253c4ef24402bdcd7ab406b2d90e in term 1.
I20260929 03:26:16.133823 15558 raft_consensus.cc:2454] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1701253c4ef24402bdcd7ab406b2d90e in term 1.
I20260929 03:26:16.134018 15232 leader_election.cc:302] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1701253c4ef24402bdcd7ab406b2d90e, b4e517794331410e873c93ca307f838d; no voters:
I20260929 03:26:16.134140 15624 raft_consensus.cc:2781] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:16.134344 15624 raft_consensus.cc:686] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [term 1 LEADER]: Becoming Leader. State: Replica: 1701253c4ef24402bdcd7ab406b2d90e, State: Running, Role: LEADER
I20260929 03:26:16.134449 15624 consensus_queue.cc:237] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } }
I20260929 03:26:16.135362 15154 catalog_manager.cc:5495] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1701253c4ef24402bdcd7ab406b2d90e (127.27.68.65). New cstate: current_term: 1 leader_uuid: "1701253c4ef24402bdcd7ab406b2d90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1701253c4ef24402bdcd7ab406b2d90e" member_type: VOTER last_known_addr { host: "127.27.68.65" port: 40681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 } health_report { overall_health: UNKNOWN } } }
I20260929 03:26:16.145669 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.143451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.27.68.65:40681)
0929 03:26:16.143514 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:16.143516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:16.143525 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:16.143566 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:16.143567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:16.143585 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:16.143646 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:16.143759 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.144222 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.144944 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:16.145146 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:16.145212 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:16.145331 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:16.145440 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:16.145499 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:16.145518 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 03:26:16.145535 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:26:16.147464 15558 raft_consensus.cc:1261] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 1 FOLLOWER]: Refusing update from remote peer 1701253c4ef24402bdcd7ab406b2d90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:16.147485 15427 raft_consensus.cc:1261] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 1 FOLLOWER]: Refusing update from remote peer 1701253c4ef24402bdcd7ab406b2d90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:16.147794 15624 consensus_queue.cc:1035] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e517794331410e873c93ca307f838d" member_type: VOTER last_known_addr { host: "127.27.68.67" port: 43769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:16.147974 15628 consensus_queue.cc:1035] T d45fec9e302e41578a200ac310f0a3e5 P 1701253c4ef24402bdcd7ab406b2d90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf61d90814ef436495b5464cdc8dcdd7" member_type: VOTER last_known_addr { host: "127.27.68.66" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:16.151427 15633 mvcc.cc:205] Tried to move back new op lower bound from 7334512132697186304 to 7334512132647362560. Current Snapshot: MvccSnapshot[applied={T|T < 7334512132697186304}]
I20260929 03:26:16.152838 15153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:16.152949 15153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:26:16.153713 15153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181fa97a8e0945d985458fe558cf72e3: Sending DeleteTablet for 3 replicas of tablet d45fec9e302e41578a200ac310f0a3e5
I20260929 03:26:16.154016 15276 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fec9e302e41578a200ac310f0a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:16 UTC) from {username='slave'} at 127.0.0.1:38574
I20260929 03:26:16.154075 15538 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fec9e302e41578a200ac310f0a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:16 UTC) from {username='slave'} at 127.0.0.1:45130
I20260929 03:26:16.154103 15407 tablet_service.cc:1514] Processing DeleteTablet for tablet d45fec9e302e41578a200ac310f0a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:26:16 UTC) from {username='slave'} at 127.0.0.1:50512
I20260929 03:26:16.154299 15642 tablet_replica.cc:330] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: stopping tablet replica
I20260929 03:26:16.154403 15642 raft_consensus.cc:2229] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:16.154508 15642 raft_consensus.cc:2258] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:16.154512 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15214
I20260929 03:26:16.154676 15641 tablet_replica.cc:330] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: stopping tablet replica
I20260929 03:26:16.155069 15641 raft_consensus.cc:2229] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:26:16.155164 15641 raft_consensus.cc:2258] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:26:16.155593 15641 ts_tablet_manager.cc:1905] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:16.155588 15642 ts_tablet_manager.cc:1905] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:26:16.157426 15642 ts_tablet_manager.cc:1918] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:16.157426 15641 ts_tablet_manager.cc:1918] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:26:16.157505 15641 log.cc:1196] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-1/wal/wals/d45fec9e302e41578a200ac310f0a3e5
I20260929 03:26:16.157505 15642 log.cc:1196] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Deleting WAL directory at /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652132259900-27921-0/minicluster-data/ts-2/wal/wals/d45fec9e302e41578a200ac310f0a3e5
I20260929 03:26:16.157765 15641 ts_tablet_manager.cc:1939] T d45fec9e302e41578a200ac310f0a3e5 P bf61d90814ef436495b5464cdc8dcdd7: Deleting consensus metadata
I20260929 03:26:16.157765 15642 ts_tablet_manager.cc:1939] T d45fec9e302e41578a200ac310f0a3e5 P b4e517794331410e873c93ca307f838d: Deleting consensus metadata
I20260929 03:26:16.158193 15141 catalog_manager.cc:4841] TS bf61d90814ef436495b5464cdc8dcdd7 (127.27.68.66:36099): tablet d45fec9e302e41578a200ac310f0a3e5 (table test-table [id=c25e83f8eb60420c9e621ecc2dced846]) successfully deleted
I20260929 03:26:16.158277 15141 catalog_manager.cc:4841] TS b4e517794331410e873c93ca307f838d (127.27.68.67:43769): tablet d45fec9e302e41578a200ac310f0a3e5 (table test-table [id=c25e83f8eb60420c9e621ecc2dced846]) successfully deleted
W20260929 03:26:16.160658 15138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.68.65:40681 (error 108)
I20260929 03:26:16.160882 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15345
I20260929 03:26:16.161545 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:16.161026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.65:40681 (local address 127.0.0.1:38594)
0929 03:26:16.161449 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:16.161503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
W20260929 03:26:16.161754 15138 catalog_manager.cc:4584] TS 1701253c4ef24402bdcd7ab406b2d90e (127.27.68.65:40681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d45fec9e302e41578a200ac310f0a3e5: Network error: Client connection negotiation failed: client connection to 127.27.68.65:40681: connect: Connection refused (error 111)
I20260929 03:26:16.166033 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15476
I20260929 03:26:16.170734 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15123
2026-09-29T03:26:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (705 ms)
[----------] 6 tests from EncryptionPolicyTest (8392 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:16 dist-test-slave-fzwk krb5kdc[15646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:16 dist-test-slave-fzwk krb5kdc[15646](info): set up 2 sockets
Sep 29 03:26:16 dist-test-slave-fzwk krb5kdc[15646](info): commencing operation
krb5kdc: starting...
W20260929 03:26:17.215708 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:17.340132 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:17.422474 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:17.422701 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:17.422725 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:17.423869 15659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:17.423905 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:17.423919 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:17.423933 15659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:17.423945 15659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:17.425382 15659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:17.425583 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:17.425762 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:17.430631 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:17.430709 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:17.430639 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:17.430850 15659 server_base.cc:1034] running on GCE node
I20260929 03:26:17.431078 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:17.431471 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:17.432662 15659 hybrid_clock.cc:648] HybridClock initialized: now 1790652377432649 us; error 48 us; skew 500 ppm
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:17.435185 15659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:17.435667 15659 webserver.cc:460] Webserver started at http://127.0.0.1:34897/ using document root <none> and password file <none>
I20260929 03:26:17.435899 15659 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:17.435967 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:17.436124 15659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:17.437197 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "3baded50584647cfbe0139f071b276c5"
format_stamp: "Formatted at 2026-09-29 03:26:17 on dist-test-slave-fzwk"
server_key: "c2ce5b62afc331d15622a2bc02b243f0"
server_key_iv: "8dad17064c4e977da9d9828e72e59bea"
server_key_version: "encryptionkey@0"
I20260929 03:26:17.437572 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "3baded50584647cfbe0139f071b276c5"
format_stamp: "Formatted at 2026-09-29 03:26:17 on dist-test-slave-fzwk"
server_key: "c2ce5b62afc331d15622a2bc02b243f0"
server_key_iv: "8dad17064c4e977da9d9828e72e59bea"
server_key_version: "encryptionkey@0"
I20260929 03:26:17.438913 15659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:17.439725 15674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:17.439917 15659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:17.440027 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "3baded50584647cfbe0139f071b276c5"
format_stamp: "Formatted at 2026-09-29 03:26:17 on dist-test-slave-fzwk"
server_key: "c2ce5b62afc331d15622a2bc02b243f0"
server_key_iv: "8dad17064c4e977da9d9828e72e59bea"
server_key_version: "encryptionkey@0"
I20260929 03:26:17.440117 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:17.453815 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:17.455886 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:17.456063 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:17.460263 15659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35513
I20260929 03:26:17.460299 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35513 every 8 connection(s)
I20260929 03:26:17.460695 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:17.461635 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:17.464131 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5: Bootstrap starting.
I20260929 03:26:17.464759 15727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:17.465070 15727 log.cc:824] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:17.465204 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15659
I20260929 03:26:17.465327 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:17.465526 27921 external_mini_cluster.cc:1428] Setting key e8e4714885e91bfb7c088896289869da
I20260929 03:26:17.465920 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5: No bootstrap required, opened a new log
I20260929 03:26:17.467762 15727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } }
I20260929 03:26:17.467938 15727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:17.467985 15727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3baded50584647cfbe0139f071b276c5, State: Initialized, Role: FOLLOWER
I20260929 03:26:17.468118 15727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } }
I20260929 03:26:17.468212 15727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:17.468258 15727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:17.468315 15727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:17.469259 15727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } }
I20260929 03:26:17.469384 15727 leader_election.cc:302] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3baded50584647cfbe0139f071b276c5; no voters:
I20260929 03:26:17.469623 15727 leader_election.cc:288] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:17.469674 15732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:17.469786 15732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [term 1 LEADER]: Becoming Leader. State: Replica: 3baded50584647cfbe0139f071b276c5, State: Running, Role: LEADER
I20260929 03:26:17.469874 15732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } }
I20260929 03:26:17.469915 15727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:17.470248 15733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3baded50584647cfbe0139f071b276c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } } }
I20260929 03:26:17.470340 15733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:17.470495 15734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3baded50584647cfbe0139f071b276c5. Latest consensus state: current_term: 1 leader_uuid: "3baded50584647cfbe0139f071b276c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3baded50584647cfbe0139f071b276c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35513 } } }
I20260929 03:26:17.470556 15734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:17.470661 15737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:17.471176 15737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:17.473179 15737 catalog_manager.cc:1347] Generated new cluster ID: 8d90597b4a654ce681aac417e3cd7473
I20260929 03:26:17.473238 15737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:26:17.478833 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:17.466073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58953 (local address 127.0.0.1:35513)
0929 03:26:17.466283 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:17.466299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:17.466328 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:26:17.467740 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:17.467748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:17.467805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:17.467959 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:17.468723 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:17.469796 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:17.470869 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:17.471639 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:17.471757 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:17.474846 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:17.474851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:17.474852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:17.474865 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:17.476970 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:17.477458 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:17.477462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:17.477463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:17.477604 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:17.477852 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:17.477853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:17.477854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:17.478224 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:17.478399 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:17.478487 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:26:17.478595 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260929 03:26:17.479974 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:17.489212 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:17.481536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53797 (local address 127.0.0.1:35513)
0929 03:26:17.481574 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:17.481576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:17.481631 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:26:17.481688 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:17.481689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:17.481699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:17.481763 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:17.482373 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:17.483160 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:17.484226 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:17.484574 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:17.484652 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:17.486185 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:17.486187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:17.486187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:17.486194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:17.487923 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:17.488359 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:17.488360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:17.488361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:17.488427 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:17.488653 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:17.488654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:17.488654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:17.488794 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:17.488918 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:17.489033 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 03:26:17.489107 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:17.503938 15737 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:17.504508 15737 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:17.509835 15737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3baded50584647cfbe0139f071b276c5: Generated new TSK 0
I20260929 03:26:17.510074 15737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:17.517074 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15659
W20260929 03:26:17.521202 15644 connection.cc:535] client connection to 127.0.0.1:35513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35513): Connection reset by peer (error 104)
2026-09-29T03:26:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15758](info): set up 2 sockets
Sep 29 03:26:17 dist-test-slave-fzwk krb5kdc[15758](info): commencing operation
krb5kdc: starting...
W20260929 03:26:18.555743 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:18 dist-test-slave-fzwk krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:18.676196 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:18.757563 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:18.757781 15771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:18.757804 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:18.758927 15771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:18.758968 15771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:18.758982 15771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:18.758996 15771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:18.759007 15771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:18.760491 15771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:18.760699 15771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:18.760891 15771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:18.765769 15776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:18.765937 15771 server_base.cc:1034] running on GCE node
W20260929 03:26:18.765769 15779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:18.765798 15777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:18.766331 15771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:18.766687 15771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:18.767858 15771 hybrid_clock.cc:648] HybridClock initialized: now 1790652378767839 us; error 41 us; skew 500 ppm
Sep 29 03:26:18 dist-test-slave-fzwk krb5kdc[15758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:18.770463 15771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:18.770977 15771 webserver.cc:460] Webserver started at http://127.0.0.1:36853/ using document root <none> and password file <none>
I20260929 03:26:18.771287 15771 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:18.771368 15771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:18.771514 15771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:18.772643 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "8c1db19dd816439dab43a312d92d6776"
format_stamp: "Formatted at 2026-09-29 03:26:18 on dist-test-slave-fzwk"
server_key: "53ac4aa267b5edb6dd52b2676669441c"
server_key_iv: "3ed749a5e5f6aab4b09f2103f93fe40e"
server_key_version: "encryptionkey@0"
I20260929 03:26:18.773036 15771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "8c1db19dd816439dab43a312d92d6776"
format_stamp: "Formatted at 2026-09-29 03:26:18 on dist-test-slave-fzwk"
server_key: "53ac4aa267b5edb6dd52b2676669441c"
server_key_iv: "3ed749a5e5f6aab4b09f2103f93fe40e"
server_key_version: "encryptionkey@0"
I20260929 03:26:18.774428 15771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:18.775177 15786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:18.775374 15771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:18.775489 15771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "8c1db19dd816439dab43a312d92d6776"
format_stamp: "Formatted at 2026-09-29 03:26:18 on dist-test-slave-fzwk"
server_key: "53ac4aa267b5edb6dd52b2676669441c"
server_key_iv: "3ed749a5e5f6aab4b09f2103f93fe40e"
server_key_version: "encryptionkey@0"
I20260929 03:26:18.775570 15771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:18.792845 15771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:18.795380 15771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:18.795550 15771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:18.799732 15771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44381
I20260929 03:26:18.799783 15838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44381 every 8 connection(s)
I20260929 03:26:18.800189 15771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:18.800937 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15771
I20260929 03:26:18.801064 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:18.801232 15839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:18.801296 27921 external_mini_cluster.cc:1428] Setting key 798660884d9fc79cf778984d4c436e36
I20260929 03:26:18.804430 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: Bootstrap starting.
I20260929 03:26:18.805207 15839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:18.805516 15839 log.cc:824] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:18.806209 15839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: No bootstrap required, opened a new log
I20260929 03:26:18.807298 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:18.801913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49581 (local address 127.0.0.1:44381)
0929 03:26:18.802076 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:18.802086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:18.802119 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:26:18.802679 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:18.802686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:18.802734 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:18.802851 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:18.803125 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:18.804075 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:18.805267 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:18.805718 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:18.805835 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:18.806594 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:18.806598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:18.806598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:18.806759 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:18.806788 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:18.806799 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:18.806862 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:18.806979 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:26:18.807024 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260929 03:26:18.808488 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:18.808991 15839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } }
I20260929 03:26:18.809181 15839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:18.809221 15839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1db19dd816439dab43a312d92d6776, State: Initialized, Role: FOLLOWER
I20260929 03:26:18.809356 15839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } }
I20260929 03:26:18.809435 15839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:18.809468 15839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:18.809526 15839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:18.810325 15839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } }
I20260929 03:26:18.810447 15839 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1db19dd816439dab43a312d92d6776; no voters:
I20260929 03:26:18.810657 15839 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:18.810717 15849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:18.810894 15849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1db19dd816439dab43a312d92d6776, State: Running, Role: LEADER
I20260929 03:26:18.810900 15839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:18.811666 15849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } }
I20260929 03:26:18.812285 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1db19dd816439dab43a312d92d6776. Latest consensus state: current_term: 1 leader_uuid: "8c1db19dd816439dab43a312d92d6776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } } }
I20260929 03:26:18.812508 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:18.812726 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c1db19dd816439dab43a312d92d6776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1db19dd816439dab43a312d92d6776" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44381 } } }
I20260929 03:26:18.812806 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776 [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:18.812857 15863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:18.812917 15863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:18.813849 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:18.810056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57615 (local address 127.0.0.1:44381)
0929 03:26:18.810115 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:18.810117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:18.810246 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:26:18.810273 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:18.810274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:18.810285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:18.810357 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:18.810507 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:18.811279 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:18.812436 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:18.812718 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:18.812763 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:18.813606 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:18.813607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:18.813608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:18.813613 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:18.813630 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:18.813634 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:18.813711 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:18.813758 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:26:18.813770 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:26:18.815363 15864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:18.815615 15864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:18.817140 15864 catalog_manager.cc:1347] Generated new cluster ID: 6932d77fecfa4913b8b0d8d2ede749b2
I20260929 03:26:18.817193 15864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:18.825603 15864 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:18.826383 15864 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:18.830176 15864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c1db19dd816439dab43a312d92d6776: Generated new TSK 0
I20260929 03:26:18.830416 15864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:18.833945 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15771
W20260929 03:26:18.838361 15756 connection.cc:535] client connection to 127.0.0.1:44381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44381): Connection reset by peer (error 104)
2026-09-29T03:26:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:18 dist-test-slave-fzwk krb5kdc[15870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:18 dist-test-slave-fzwk krb5kdc[15870](info): set up 2 sockets
Sep 29 03:26:18 dist-test-slave-fzwk krb5kdc[15870](info): commencing operation
krb5kdc: starting...
W20260929 03:26:19.859630 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:19 dist-test-slave-fzwk krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:19.978263 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:20.060043 15883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:20.060315 15883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:20.060356 15883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:20.061597 15883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:20.061671 15883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:20.061697 15883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:20.061717 15883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:20.061744 15883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:20.063409 15883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:20.063705 15883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:20.063939 15883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:20.068907 15888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:20.069008 15883 server_base.cc:1034] running on GCE node
W20260929 03:26:20.068886 15889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:20.068886 15891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:20.069370 15883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:20.069679 15883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:20.070827 15883 hybrid_clock.cc:648] HybridClock initialized: now 1790652380070806 us; error 37 us; skew 500 ppm
Sep 29 03:26:20 dist-test-slave-fzwk krb5kdc[15870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:20.078806 15883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:20.079396 15883 webserver.cc:460] Webserver started at http://127.0.0.1:44445/ using document root <none> and password file <none>
I20260929 03:26:20.079645 15883 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:20.079715 15883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:20.079879 15883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:20.080997 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "83f167a5676049f1a9e5ca27a46a2328"
format_stamp: "Formatted at 2026-09-29 03:26:20 on dist-test-slave-fzwk"
server_key: "3ee56f24cf88914d8389ea41beb83d53"
server_key_iv: "7077f61a9d0294b278907ba216ce60b3"
server_key_version: "encryptionkey@0"
I20260929 03:26:20.081378 15883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "83f167a5676049f1a9e5ca27a46a2328"
format_stamp: "Formatted at 2026-09-29 03:26:20 on dist-test-slave-fzwk"
server_key: "3ee56f24cf88914d8389ea41beb83d53"
server_key_iv: "7077f61a9d0294b278907ba216ce60b3"
server_key_version: "encryptionkey@0"
I20260929 03:26:20.082721 15883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:26:20.083567 15898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:20.083868 15883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:20.083976 15883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "83f167a5676049f1a9e5ca27a46a2328"
format_stamp: "Formatted at 2026-09-29 03:26:20 on dist-test-slave-fzwk"
server_key: "3ee56f24cf88914d8389ea41beb83d53"
server_key_iv: "7077f61a9d0294b278907ba216ce60b3"
server_key_version: "encryptionkey@0"
I20260929 03:26:20.084057 15883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:20.087387 15883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:20.089398 15883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:20.089570 15883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:20.093873 15883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43009
I20260929 03:26:20.093904 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43009 every 8 connection(s)
I20260929 03:26:20.094264 15883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:20.094923 15951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:20.097258 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328: Bootstrap starting.
I20260929 03:26:20.097810 15951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:20.098057 15951 log.cc:824] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:20.098723 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328: No bootstrap required, opened a new log
I20260929 03:26:20.100507 15951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260929 03:26:20.100660 15951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:20.100682 15951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f167a5676049f1a9e5ca27a46a2328, State: Initialized, Role: FOLLOWER
I20260929 03:26:20.100824 15951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260929 03:26:20.100903 15951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:20.100926 15951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:20.100960 15951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:20.101497 15951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260929 03:26:20.101583 15951 leader_election.cc:302] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83f167a5676049f1a9e5ca27a46a2328; no voters:
I20260929 03:26:20.101711 15951 leader_election.cc:288] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:20.101795 15954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:20.101955 15954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [term 1 LEADER]: Becoming Leader. State: Replica: 83f167a5676049f1a9e5ca27a46a2328, State: Running, Role: LEADER
I20260929 03:26:20.102015 15951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:20.102083 15954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } }
I20260929 03:26:20.102522 15956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83f167a5676049f1a9e5ca27a46a2328. Latest consensus state: current_term: 1 leader_uuid: "83f167a5676049f1a9e5ca27a46a2328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } } }
I20260929 03:26:20.102662 15956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:20.102692 15955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83f167a5676049f1a9e5ca27a46a2328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83f167a5676049f1a9e5ca27a46a2328" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43009 } } }
I20260929 03:26:20.102849 15955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:20.103104 15963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:20.103415 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15883
I20260929 03:26:20.103564 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:20.103634 15963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:20.103775 27921 external_mini_cluster.cc:1428] Setting key 14cf450ee5a2bb67a9a3c06b94921779
I20260929 03:26:20.105334 15963 catalog_manager.cc:1347] Generated new cluster ID: afdfda4cf93d401b965312eaf048fd9e
I20260929 03:26:20.105381 15963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:20.106159 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:20.104505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40645 (local address 127.0.0.1:43009)
0929 03:26:20.104667 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:20.104678 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:20.104714 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:26:20.104828 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:20.104833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:20.104867 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:20.105010 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:20.105760 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:20.105765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:20.105765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:20.105769 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:20.105812 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:20.105823 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:20.105892 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:20.105967 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:26:20.106011 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260929 03:26:20.111665 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:20.116936 15963 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:20.117121 15963 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:20.117805 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:20.113155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49317 (local address 127.0.0.1:43009)
0929 03:26:20.113194 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:20.113196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:20.116634 (+ 3438us) server_negotiation.cc:407] Connection header received
0929 03:26:20.116741 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:20.116742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:20.116749 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:20.116814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:20.117561 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:20.117562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:20.117563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:20.117567 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:20.117583 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:20.117587 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:20.117645 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:20.117702 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:26:20.117718 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:26:20.128340 15963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83f167a5676049f1a9e5ca27a46a2328: Generated new TSK 0
I20260929 03:26:20.128584 15963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:20.132282 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15883
2026-09-29T03:26:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:20 dist-test-slave-fzwk krb5kdc[15982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:20 dist-test-slave-fzwk krb5kdc[15982](info): set up 2 sockets
Sep 29 03:26:20 dist-test-slave-fzwk krb5kdc[15982](info): commencing operation
krb5kdc: starting...
W20260929 03:26:21.167631 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[15982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:21.290076 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:21.378307 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:21.378594 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:21.378643 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:21.379901 15995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:21.379980 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:21.380008 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:21.380033 15995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:21.380065 15995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:21.381631 15995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:21.381973 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:21.382228 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:21.387416 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:21.387462 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:21.387418 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:21.387651 15995 server_base.cc:1034] running on GCE node
I20260929 03:26:21.387952 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:21.388257 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:21.389508 15995 hybrid_clock.cc:648] HybridClock initialized: now 1790652381389490 us; error 25 us; skew 500 ppm
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[15982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:21.392210 15995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:21.392728 15995 webserver.cc:460] Webserver started at http://127.0.0.1:42745/ using document root <none> and password file <none>
I20260929 03:26:21.392976 15995 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:21.393026 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:21.393213 15995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:21.394335 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0334d45e9ca546aeaac572f35b0dc547"
format_stamp: "Formatted at 2026-09-29 03:26:21 on dist-test-slave-fzwk"
server_key: "13c4c5f7a64a61db9223475b132874dd"
server_key_iv: "30ceca3dc4848749f3e37b5599a9ba44"
server_key_version: "encryptionkey@0"
I20260929 03:26:21.394709 15995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0334d45e9ca546aeaac572f35b0dc547"
format_stamp: "Formatted at 2026-09-29 03:26:21 on dist-test-slave-fzwk"
server_key: "13c4c5f7a64a61db9223475b132874dd"
server_key_iv: "30ceca3dc4848749f3e37b5599a9ba44"
server_key_version: "encryptionkey@0"
I20260929 03:26:21.396246 15995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:21.397023 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:21.397219 15995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:21.397316 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0334d45e9ca546aeaac572f35b0dc547"
format_stamp: "Formatted at 2026-09-29 03:26:21 on dist-test-slave-fzwk"
server_key: "13c4c5f7a64a61db9223475b132874dd"
server_key_iv: "30ceca3dc4848749f3e37b5599a9ba44"
server_key_version: "encryptionkey@0"
I20260929 03:26:21.397392 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:21.414860 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:21.416821 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:21.416975 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:21.421149 15995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36051
I20260929 03:26:21.421183 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36051 every 8 connection(s)
I20260929 03:26:21.421566 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:21.422441 16063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:21.424810 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: Bootstrap starting.
I20260929 03:26:21.425091 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 15995
I20260929 03:26:21.425217 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:21.425413 27921 external_mini_cluster.cc:1428] Setting key 39eeefdd8c604bf1b8096d7139025ef7
I20260929 03:26:21.425588 16063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:21.425928 16063 log.cc:824] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:21.426676 16063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: No bootstrap required, opened a new log
I20260929 03:26:21.429075 16063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260929 03:26:21.429257 16063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:21.429292 16063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0334d45e9ca546aeaac572f35b0dc547, State: Initialized, Role: FOLLOWER
I20260929 03:26:21.429411 16063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260929 03:26:21.429482 16063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:21.429510 16063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:21.429553 16063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:21.430193 16063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260929 03:26:21.430333 16063 leader_election.cc:302] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0334d45e9ca546aeaac572f35b0dc547; no voters:
I20260929 03:26:21.430523 16063 leader_election.cc:288] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:21.430567 16068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:21.430766 16068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [term 1 LEADER]: Becoming Leader. State: Replica: 0334d45e9ca546aeaac572f35b0dc547, State: Running, Role: LEADER
I20260929 03:26:21.430832 16063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:21.430909 16068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } }
I20260929 03:26:21.431393 16069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0334d45e9ca546aeaac572f35b0dc547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } } }
I20260929 03:26:21.431618 16069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:21.431542 16070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0334d45e9ca546aeaac572f35b0dc547. Latest consensus state: current_term: 1 leader_uuid: "0334d45e9ca546aeaac572f35b0dc547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0334d45e9ca546aeaac572f35b0dc547" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36051 } } }
I20260929 03:26:21.432075 16070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547 [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:21.432721 16084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:21.432782 16084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:21.432832 16077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:21.433017 16077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[15982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:26:21.434530 16077 catalog_manager.cc:1347] Generated new cluster ID: 0194b35035514837bef97c34c557afc1
I20260929 03:26:21.434589 16077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:21.437335 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:21.425994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42319 (local address 127.0.0.1:36051)
0929 03:26:21.426156 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:21.426168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:21.426207 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:26:21.426768 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:21.426778 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:21.426841 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:21.426960 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:21.427809 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:21.428698 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:21.429761 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:21.430178 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:21.430266 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:21.433665 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:21.433670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:21.433670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:21.433683 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:21.435690 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:21.436095 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:21.436098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:21.436099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:21.436208 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:21.436407 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:21.436408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:21.436409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:21.436737 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:21.436906 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:21.436977 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:26:21.437115 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260929 03:26:21.438378 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:21.447182 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:21.440052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51667 (local address 127.0.0.1:36051)
0929 03:26:21.440087 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:21.440089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:21.440098 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:21.440172 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:21.440172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:21.440181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:21.440224 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:21.440868 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:21.441618 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:21.442606 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:21.442895 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:21.442952 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:21.444357 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:21.444358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:21.444358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:21.444365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:21.445948 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:21.446345 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:21.446346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:21.446346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:21.446400 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:21.446612 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:21.446613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:21.446613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:21.446725 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:21.446807 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:21.446929 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:26:21.446999 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:21.453626 16077 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:21.454368 16077 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:21.458968 16077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0334d45e9ca546aeaac572f35b0dc547: Generated new TSK 0
I20260929 03:26:21.459200 16077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:21.468851 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 15995
2026-09-29T03:26:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[16094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[16094](info): set up 2 sockets
Sep 29 03:26:21 dist-test-slave-fzwk krb5kdc[16094](info): commencing operation
krb5kdc: starting...
W20260929 03:26:22.511565 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:22 dist-test-slave-fzwk krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:22.634573 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:22.718228 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:22.718488 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:22.718529 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:22.719933 16107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:22.720011 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:22.720036 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:22.720073 16107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:22.720119 16107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:22.721655 16107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:22.721943 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:22.722180 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:26:22.727171 16107 server_base.cc:1034] running on GCE node
W20260929 03:26:22.727137 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:22.727146 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:22.727136 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:22.727661 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:22.727977 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:22.729151 16107 hybrid_clock.cc:648] HybridClock initialized: now 1790652382729143 us; error 37 us; skew 500 ppm
Sep 29 03:26:22 dist-test-slave-fzwk krb5kdc[16094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:22.731786 16107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:22.732259 16107 webserver.cc:460] Webserver started at http://127.0.0.1:41059/ using document root <none> and password file <none>
I20260929 03:26:22.732446 16107 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:22.732486 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:22.732579 16107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:22.733608 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "c1da6d2be9674906acc0c92cbe752895"
format_stamp: "Formatted at 2026-09-29 03:26:22 on dist-test-slave-fzwk"
server_key: "b30a97f9c7ffc9d14ed9d2d4ec0b16bc"
server_key_iv: "3e3d36a881e4cba1784812d5e01f28e8"
server_key_version: "encryptionkey@0"
I20260929 03:26:22.733920 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "c1da6d2be9674906acc0c92cbe752895"
format_stamp: "Formatted at 2026-09-29 03:26:22 on dist-test-slave-fzwk"
server_key: "b30a97f9c7ffc9d14ed9d2d4ec0b16bc"
server_key_iv: "3e3d36a881e4cba1784812d5e01f28e8"
server_key_version: "encryptionkey@0"
I20260929 03:26:22.735263 16107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:22.735996 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:22.736181 16107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:22.736284 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "c1da6d2be9674906acc0c92cbe752895"
format_stamp: "Formatted at 2026-09-29 03:26:22 on dist-test-slave-fzwk"
server_key: "b30a97f9c7ffc9d14ed9d2d4ec0b16bc"
server_key_iv: "3e3d36a881e4cba1784812d5e01f28e8"
server_key_version: "encryptionkey@0"
I20260929 03:26:22.736356 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:22.744082 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:22.746155 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:22.746284 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:22.750525 16107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43737
I20260929 03:26:22.750605 16174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43737 every 8 connection(s)
I20260929 03:26:22.750998 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:22.752005 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:22.754339 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895: Bootstrap starting.
I20260929 03:26:22.754884 16175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:22.755133 16175 log.cc:824] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:22.755921 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895: No bootstrap required, opened a new log
I20260929 03:26:22.757477 16175 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } }
I20260929 03:26:22.757625 16175 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:22.757648 16175 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1da6d2be9674906acc0c92cbe752895, State: Initialized, Role: FOLLOWER
I20260929 03:26:22.757757 16175 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } }
I20260929 03:26:22.757845 16175 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:22.757885 16175 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:22.757941 16175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:22.758551 16175 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } }
I20260929 03:26:22.758688 16175 leader_election.cc:302] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1da6d2be9674906acc0c92cbe752895; no voters:
I20260929 03:26:22.758903 16175 leader_election.cc:288] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:22.758957 16178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:22.759181 16178 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [term 1 LEADER]: Becoming Leader. State: Replica: c1da6d2be9674906acc0c92cbe752895, State: Running, Role: LEADER
I20260929 03:26:22.759184 16175 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:22.759397 16178 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } }
I20260929 03:26:22.759543 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16107
I20260929 03:26:22.759644 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:22.759846 27921 external_mini_cluster.cc:1428] Setting key 9920bdd3edd5e3fb64f3f8fec6213c96
I20260929 03:26:22.759835 16179 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1da6d2be9674906acc0c92cbe752895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } } }
I20260929 03:26:22.759931 16179 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:22.760087 16180 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1da6d2be9674906acc0c92cbe752895. Latest consensus state: current_term: 1 leader_uuid: "c1da6d2be9674906acc0c92cbe752895" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1da6d2be9674906acc0c92cbe752895" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43737 } } }
I20260929 03:26:22.760151 16180 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:22.760675 16182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:22.761299 16182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:22.763166 16182 catalog_manager.cc:1347] Generated new cluster ID: 29789e858c70496a8cb8560174f5aa04
I20260929 03:26:22.763270 16182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:22.766783 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:22.760566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47461 (local address 127.0.0.1:43737)
0929 03:26:22.760961 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:22.760972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:22.761013 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:26:22.761152 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:22.761161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:22.761226 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:22.761671 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:22.761770 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:22.763154 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:22.764400 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:22.765102 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:22.765241 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:22.766043 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:22.766048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:22.766048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:22.766267 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:22.766309 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:22.766325 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:22.766404 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:22.766477 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:26:22.766547 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":92,"threads_started":1}
I20260929 03:26:22.768054 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:22.773368 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:22.769640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40423 (local address 127.0.0.1:43737)
0929 03:26:22.769687 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:22.769689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:22.769850 (+ 161us) server_negotiation.cc:407] Connection header received
0929 03:26:22.769901 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:22.769902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:22.769914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:22.769987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:22.770133 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:22.770872 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:22.771953 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:22.772274 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:22.772342 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:22.773106 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:22.773107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:22.773108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:22.773113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:22.773128 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:22.773133 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:22.773198 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:22.773258 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:26:22.773286 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:26:22.776396 16182 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:22.776954 16182 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:22.781093 16182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1da6d2be9674906acc0c92cbe752895: Generated new TSK 0
I20260929 03:26:22.781302 16182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:22.785122 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16107
2026-09-29T03:26:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:22 dist-test-slave-fzwk krb5kdc[16206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:22 dist-test-slave-fzwk krb5kdc[16206](info): set up 2 sockets
Sep 29 03:26:22 dist-test-slave-fzwk krb5kdc[16206](info): commencing operation
krb5kdc: starting...
W20260929 03:26:23.823602 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:23 dist-test-slave-fzwk krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:23.942015 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:24.024101 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:24.024322 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:24.024344 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:24.025441 16219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:24.025475 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:24.025489 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:24.025502 16219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:24.025514 16219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:24.026901 16219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:24.027097 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:24.027307 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:24.032218 16227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:24.032264 16219 server_base.cc:1034] running on GCE node
W20260929 03:26:24.032233 16225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:24.032225 16224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:24.032713 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:24.033030 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:24.034201 16219 hybrid_clock.cc:648] HybridClock initialized: now 1790652384034210 us; error 49 us; skew 500 ppm
Sep 29 03:26:24 dist-test-slave-fzwk krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:24.036743 16219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:24.037217 16219 webserver.cc:460] Webserver started at http://127.0.0.1:43857/ using document root <none> and password file <none>
I20260929 03:26:24.037411 16219 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:24.037452 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:24.037541 16219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:24.038578 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0c06aab895a54cb68a9f9e12c5d301d0"
format_stamp: "Formatted at 2026-09-29 03:26:24 on dist-test-slave-fzwk"
server_key: "698c6b4c175fb447bc5705844712faee"
server_key_iv: "6a0f32722a56fbef24fa9a6c12fc4281"
server_key_version: "encryptionkey@0"
I20260929 03:26:24.038894 16219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0c06aab895a54cb68a9f9e12c5d301d0"
format_stamp: "Formatted at 2026-09-29 03:26:24 on dist-test-slave-fzwk"
server_key: "698c6b4c175fb447bc5705844712faee"
server_key_iv: "6a0f32722a56fbef24fa9a6c12fc4281"
server_key_version: "encryptionkey@0"
I20260929 03:26:24.040339 16219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:24.041059 16234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:24.041251 16219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:24.041326 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0c06aab895a54cb68a9f9e12c5d301d0"
format_stamp: "Formatted at 2026-09-29 03:26:24 on dist-test-slave-fzwk"
server_key: "698c6b4c175fb447bc5705844712faee"
server_key_iv: "6a0f32722a56fbef24fa9a6c12fc4281"
server_key_version: "encryptionkey@0"
I20260929 03:26:24.041409 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:24.044848 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:24.047502 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:24.047617 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:24.051589 16219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38693
I20260929 03:26:24.051625 16286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38693 every 8 connection(s)
I20260929 03:26:24.051999 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:24.052995 16287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:24.055410 16287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0: Bootstrap starting.
I20260929 03:26:24.056005 16287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:24.056298 16287 log.cc:824] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:24.056625 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16219
I20260929 03:26:24.056747 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:24.056949 27921 external_mini_cluster.cc:1428] Setting key 43a641663d759e6d967d2fae6d38d0c4
I20260929 03:26:24.057149 16287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0: No bootstrap required, opened a new log
I20260929 03:26:24.059639 16287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260929 03:26:24.059818 16287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:24.059865 16287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c06aab895a54cb68a9f9e12c5d301d0, State: Initialized, Role: FOLLOWER
I20260929 03:26:24.059998 16287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260929 03:26:24.060024 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:24.057508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37265 (local address 127.0.0.1:38693)
0929 03:26:24.057709 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:24.057725 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:24.057753 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:26:24.058512 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:24.058519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:24.058547 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:24.058669 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:24.059322 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:24.059327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:24.059327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:24.059332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:24.059385 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:24.059396 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:24.059453 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:24.059545 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:26:24.059600 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260929 03:26:24.060105 16287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:24.060151 16287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:24.060207 16287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:24.060998 16287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260929 03:26:24.061117 16287 leader_election.cc:302] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c06aab895a54cb68a9f9e12c5d301d0; no voters:
I20260929 03:26:24.061224 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:24.061285 16287 leader_election.cc:288] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:24.061390 16292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:24.061522 16287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:24.061581 16292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c06aab895a54cb68a9f9e12c5d301d0, State: Running, Role: LEADER
I20260929 03:26:24.061708 16292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } }
I20260929 03:26:24.062142 16294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c06aab895a54cb68a9f9e12c5d301d0. Latest consensus state: current_term: 1 leader_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } } }
I20260929 03:26:24.062134 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c06aab895a54cb68a9f9e12c5d301d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38693 } } }
I20260929 03:26:24.062247 16294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:24.062253 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:24.062500 16302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:24.063087 16302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:24.064684 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:24.062872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37099 (local address 127.0.0.1:38693)
0929 03:26:24.062909 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:24.062911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:24.063284 (+ 373us) server_negotiation.cc:407] Connection header received
0929 03:26:24.063454 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:24.063455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:24.063460 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:24.063491 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:24.064428 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:24.064429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:24.064430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:24.064434 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:24.064450 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:24.064454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:24.064498 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:24.064607 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:26:24.064623 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:24.065099 16302 catalog_manager.cc:1347] Generated new cluster ID: b351e34383864abf8beb905937933a69
I20260929 03:26:24.065184 16302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:24.081624 16302 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:24.081851 16302 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:24.085270 16302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c06aab895a54cb68a9f9e12c5d301d0: Generated new TSK 0
I20260929 03:26:24.085486 16302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:24.088333 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16219
2026-09-29T03:26:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:24 dist-test-slave-fzwk krb5kdc[16318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:24 dist-test-slave-fzwk krb5kdc[16318](info): set up 2 sockets
Sep 29 03:26:24 dist-test-slave-fzwk krb5kdc[16318](info): commencing operation
krb5kdc: starting...
W20260929 03:26:25.135673 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:25.256002 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:25.340338 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:25.340591 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:25.340634 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:25.341925 16331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:25.341996 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:25.342025 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:25.342063 16331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:25.342092 16331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:25.343751 16331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:25.344046 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:25.344290 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:25.349143 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:25.349145 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:25.349277 16331 server_base.cc:1034] running on GCE node
W20260929 03:26:25.349216 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:25.349627 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:25.349888 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:25.351034 16331 hybrid_clock.cc:648] HybridClock initialized: now 1790652385351021 us; error 29 us; skew 500 ppm
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:25.362507 16331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:25.363061 16331 webserver.cc:460] Webserver started at http://127.0.0.1:39395/ using document root <none> and password file <none>
I20260929 03:26:25.363365 16331 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:25.363415 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:25.363574 16331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:25.364717 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "a80774524cac4996b6c3d429e7e2fe68"
format_stamp: "Formatted at 2026-09-29 03:26:25 on dist-test-slave-fzwk"
server_key: "4aafae153373f3fca464ba60d611b161"
server_key_iv: "6c1dd37299df7e7638aaf23d8e2d69e5"
server_key_version: "encryptionkey@0"
I20260929 03:26:25.365127 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "a80774524cac4996b6c3d429e7e2fe68"
format_stamp: "Formatted at 2026-09-29 03:26:25 on dist-test-slave-fzwk"
server_key: "4aafae153373f3fca464ba60d611b161"
server_key_iv: "6c1dd37299df7e7638aaf23d8e2d69e5"
server_key_version: "encryptionkey@0"
I20260929 03:26:25.366515 16331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:26:25.367317 16346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:25.367515 16331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:25.367619 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "a80774524cac4996b6c3d429e7e2fe68"
format_stamp: "Formatted at 2026-09-29 03:26:25 on dist-test-slave-fzwk"
server_key: "4aafae153373f3fca464ba60d611b161"
server_key_iv: "6c1dd37299df7e7638aaf23d8e2d69e5"
server_key_version: "encryptionkey@0"
I20260929 03:26:25.367702 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:25.379333 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:25.381428 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:25.381597 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:25.385985 16331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42067
I20260929 03:26:25.386019 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42067 every 8 connection(s)
I20260929 03:26:25.386417 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:25.387400 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:25.389799 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68: Bootstrap starting.
I20260929 03:26:25.390375 16399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:25.390628 16399 log.cc:824] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:25.391341 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16331
I20260929 03:26:25.391467 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:25.391580 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68: No bootstrap required, opened a new log
I20260929 03:26:25.391669 27921 external_mini_cluster.cc:1428] Setting key 6085843f1959d9d68e4e904afc3b9b4b
I20260929 03:26:25.394147 16399 raft_consensus.cc:348] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260929 03:26:25.394299 16399 raft_consensus.cc:374] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:25.394321 16399 raft_consensus.cc:729] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80774524cac4996b6c3d429e7e2fe68, State: Initialized, Role: FOLLOWER
I20260929 03:26:25.394459 16399 consensus_queue.cc:260] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260929 03:26:25.394532 16399 raft_consensus.cc:388] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:25.394582 16399 raft_consensus.cc:482] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:25.394632 16399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:25.395331 16399 raft_consensus.cc:504] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260929 03:26:25.395444 16399 leader_election.cc:302] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a80774524cac4996b6c3d429e7e2fe68; no voters:
I20260929 03:26:25.395640 16399 leader_election.cc:288] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:25.395668 16404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:25.395840 16404 raft_consensus.cc:686] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [term 1 LEADER]: Becoming Leader. State: Replica: a80774524cac4996b6c3d429e7e2fe68, State: Running, Role: LEADER
I20260929 03:26:25.395992 16399 sys_catalog.cc:564] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:25.395991 16404 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } }
I20260929 03:26:25.396560 16406 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a80774524cac4996b6c3d429e7e2fe68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260929 03:26:25.396660 16406 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:25.396907 16405 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a80774524cac4996b6c3d429e7e2fe68. Latest consensus state: current_term: 1 leader_uuid: "a80774524cac4996b6c3d429e7e2fe68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80774524cac4996b6c3d429e7e2fe68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42067 } } }
I20260929 03:26:25.397282 16405 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:25.397444 16413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:25.398157 16413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:25.400111 16413 catalog_manager.cc:1347] Generated new cluster ID: 5e296e67254344b4b9f7c9fc204f5943
I20260929 03:26:25.400172 16413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:26:25.405079 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:25.392305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:44709 (local address 127.0.0.1:42067)
0929 03:26:25.392494 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:25.392504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:25.392544 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:26:25.393030 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:25.393038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:25.393094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:25.393223 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:25.394048 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:25.395603 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:25.396743 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:25.397402 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:25.397499 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:25.400904 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:25.400909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:25.400910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:25.400924 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:25.403024 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:25.403578 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:25.403581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:25.403582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:25.403694 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:25.403954 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:25.403954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:25.403955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:25.404265 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:25.404422 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:25.404722 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:26:25.404847 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260929 03:26:25.406308 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:25.419621 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:25.411596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:43273 (local address 127.0.0.1:42067)
0929 03:26:25.411638 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:25.411640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:25.411703 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:26:25.411775 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:25.411776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:25.411788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:25.411866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:25.412588 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:25.413389 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:25.414482 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:25.414835 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:25.414910 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:25.416544 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:25.416545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:25.416545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:25.416553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:25.418225 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:25.418701 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:25.418702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:25.418702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:25.418769 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:25.419011 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:25.419012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:25.419012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:25.419147 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:25.419300 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:25.419443 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:26:25.419514 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:25.427976 16413 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:25.428582 16413 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:25.437889 16413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a80774524cac4996b6c3d429e7e2fe68: Generated new TSK 0
I20260929 03:26:25.438123 16413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:25.441478 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16331
W20260929 03:26:25.446055 16316 connection.cc:535] client connection to 127.0.0.1:42067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42067): Connection reset by peer (error 104)
2026-09-29T03:26:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16430](info): set up 2 sockets
Sep 29 03:26:25 dist-test-slave-fzwk krb5kdc[16430](info): commencing operation
krb5kdc: starting...
W20260929 03:26:26.475910 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:26 dist-test-slave-fzwk krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:26.597779 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:26.680415 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:26.680636 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:26.680660 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:26.681855 16443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:26.681901 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:26.681915 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:26.681928 16443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:26.681981 16443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:26.683614 16443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:26.683883 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:26.684136 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:26.689659 16449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:26.689692 16451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:26.689716 16448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:26.689929 16443 server_base.cc:1034] running on GCE node
I20260929 03:26:26.690160 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:26.690449 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:26.691614 16443 hybrid_clock.cc:648] HybridClock initialized: now 1790652386691600 us; error 47 us; skew 500 ppm
Sep 29 03:26:26 dist-test-slave-fzwk krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:26.694130 16443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:26.694629 16443 webserver.cc:460] Webserver started at http://127.0.0.1:35199/ using document root <none> and password file <none>
I20260929 03:26:26.694886 16443 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:26.694957 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:26.695107 16443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:26.696231 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "b27fe2e8512c48ef8cc0740a38f89004"
format_stamp: "Formatted at 2026-09-29 03:26:26 on dist-test-slave-fzwk"
server_key: "fbc11995f48534b06c00e267244a1d62"
server_key_iv: "320666102f0140460151a661d7d2752d"
server_key_version: "encryptionkey@0"
I20260929 03:26:26.696616 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "b27fe2e8512c48ef8cc0740a38f89004"
format_stamp: "Formatted at 2026-09-29 03:26:26 on dist-test-slave-fzwk"
server_key: "fbc11995f48534b06c00e267244a1d62"
server_key_iv: "320666102f0140460151a661d7d2752d"
server_key_version: "encryptionkey@0"
I20260929 03:26:26.698036 16443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:26:26.698796 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:26.698974 16443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:26.699100 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "b27fe2e8512c48ef8cc0740a38f89004"
format_stamp: "Formatted at 2026-09-29 03:26:26 on dist-test-slave-fzwk"
server_key: "fbc11995f48534b06c00e267244a1d62"
server_key_iv: "320666102f0140460151a661d7d2752d"
server_key_version: "encryptionkey@0"
I20260929 03:26:26.699177 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:26.729012 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:26.731258 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:26.731443 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:26.735721 16443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44193
I20260929 03:26:26.735759 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44193 every 8 connection(s)
I20260929 03:26:26.736147 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:26.737068 16511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:26.739437 16511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004: Bootstrap starting.
I20260929 03:26:26.739986 16511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:26.740232 16511 log.cc:824] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:26.740906 16511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004: No bootstrap required, opened a new log
I20260929 03:26:26.742506 16511 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } }
I20260929 03:26:26.742653 16511 raft_consensus.cc:374] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:26.742676 16511 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27fe2e8512c48ef8cc0740a38f89004, State: Initialized, Role: FOLLOWER
I20260929 03:26:26.742781 16511 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } }
I20260929 03:26:26.742834 16511 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:26.742856 16511 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:26.742933 16511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:26.743317 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16443
I20260929 03:26:26.743436 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:26.743650 27921 external_mini_cluster.cc:1428] Setting key d1eb33bfdeaf1e9a462ac84d0e603748
I20260929 03:26:26.743672 16511 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } }
I20260929 03:26:26.743801 16511 leader_election.cc:302] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27fe2e8512c48ef8cc0740a38f89004; no voters:
I20260929 03:26:26.743980 16511 leader_election.cc:288] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:26.744022 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:26.744273 16514 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [term 1 LEADER]: Becoming Leader. State: Replica: b27fe2e8512c48ef8cc0740a38f89004, State: Running, Role: LEADER
I20260929 03:26:26.744352 16511 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:26.744524 16514 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } }
I20260929 03:26:26.745051 16518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b27fe2e8512c48ef8cc0740a38f89004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } } }
I20260929 03:26:26.745183 16518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:26.745493 16522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:26.745123 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27fe2e8512c48ef8cc0740a38f89004. Latest consensus state: current_term: 1 leader_uuid: "b27fe2e8512c48ef8cc0740a38f89004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27fe2e8512c48ef8cc0740a38f89004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44193 } } }
I20260929 03:26:26.745919 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:26.746376 16522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:26.748080 16522 catalog_manager.cc:1347] Generated new cluster ID: c062cfdd30ab4dc5b1b60ca65559d5de
I20260929 03:26:26.748138 16522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:26.749987 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:26.744154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:48845 (local address 127.0.0.1:44193)
0929 03:26:26.744468 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:26.744477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:26.744660 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:26:26.744809 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:26.744815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:26.744863 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:26.745007 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:26.745218 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:26.746754 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:26.747880 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:26.748443 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:26.748573 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:26.749320 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:26.749324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:26.749324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:26.749527 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:26.749574 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:26.749585 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:26.749682 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:26.749765 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:26:26.749796 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":63,"threads_started":1}
I20260929 03:26:26.750950 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:26.755528 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:26.752447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:59963 (local address 127.0.0.1:44193)
0929 03:26:26.752486 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:26.752488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:26.752619 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:26:26.752696 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:26.752697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:26.752707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:26.752763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:26.752918 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:26.753426 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:26.754157 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:26.754384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:26.754457 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:26.755172 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:26.755173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:26.755173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:26.755178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:26.755192 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:26.755196 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:26.755296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:26.755416 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:26:26.755430 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:26.759407 16522 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:26.759919 16522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:26.767393 16522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b27fe2e8512c48ef8cc0740a38f89004: Generated new TSK 0
I20260929 03:26:26.767622 16522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:26.772827 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16443
W20260929 03:26:26.777163 16428 connection.cc:535] client connection to 127.0.0.1:44193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44193): Connection reset by peer (error 104)
2026-09-29T03:26:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:26 dist-test-slave-fzwk krb5kdc[16542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:26 dist-test-slave-fzwk krb5kdc[16542](info): set up 2 sockets
Sep 29 03:26:26 dist-test-slave-fzwk krb5kdc[16542](info): commencing operation
krb5kdc: starting...
W20260929 03:26:27.791783 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:27 dist-test-slave-fzwk krb5kdc[16542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:27.911015 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:27.994724 16555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:27.994992 16555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:27.995033 16555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:27.996294 16555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:27.996369 16555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:27.996393 16555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:27.996417 16555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:27.996467 16555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:27.998059 16555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:27.998361 16555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:27.998612 16555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:28.003633 16561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:28.003646 16563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:28.003708 16555 server_base.cc:1034] running on GCE node
W20260929 03:26:28.003618 16560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:28.004107 16555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:28.004441 16555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:28.005615 16555 hybrid_clock.cc:648] HybridClock initialized: now 1790652388005611 us; error 36 us; skew 500 ppm
Sep 29 03:26:28 dist-test-slave-fzwk krb5kdc[16542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:28.008313 16555 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:28.008785 16555 webserver.cc:460] Webserver started at http://127.0.0.1:42325/ using document root <none> and password file <none>
I20260929 03:26:28.009037 16555 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:28.009114 16555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:28.009264 16555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:28.010375 16555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "f057c6da08ce434f91e39eceae63f364"
format_stamp: "Formatted at 2026-09-29 03:26:28 on dist-test-slave-fzwk"
server_key: "13c28ab0e03e4c43d079ea1a35fafa96"
server_key_iv: "a33b24b1c45b88f77180f866d8438417"
server_key_version: "encryptionkey@0"
I20260929 03:26:28.010759 16555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "f057c6da08ce434f91e39eceae63f364"
format_stamp: "Formatted at 2026-09-29 03:26:28 on dist-test-slave-fzwk"
server_key: "13c28ab0e03e4c43d079ea1a35fafa96"
server_key_iv: "a33b24b1c45b88f77180f866d8438417"
server_key_version: "encryptionkey@0"
I20260929 03:26:28.012283 16555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:28.013087 16570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:28.013284 16555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:26:28.013413 16555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "f057c6da08ce434f91e39eceae63f364"
format_stamp: "Formatted at 2026-09-29 03:26:28 on dist-test-slave-fzwk"
server_key: "13c28ab0e03e4c43d079ea1a35fafa96"
server_key_iv: "a33b24b1c45b88f77180f866d8438417"
server_key_version: "encryptionkey@0"
I20260929 03:26:28.013504 16555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:28.017021 16555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:28.019405 16555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:28.019568 16555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:28.023716 16555 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45127
I20260929 03:26:28.023751 16622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45127 every 8 connection(s)
I20260929 03:26:28.024107 16555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:28.025094 16623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:28.026147 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16555
I20260929 03:26:28.026274 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:28.026460 27921 external_mini_cluster.cc:1428] Setting key 39e8a09aca146669fa53c0301fd0d0bc
I20260929 03:26:28.027781 16623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364: Bootstrap starting.
I20260929 03:26:28.028553 16623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:28.028865 16623 log.cc:824] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:28.029109 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:28.027030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45283 (local address 127.0.0.1:45127)
0929 03:26:28.027207 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:28.027254 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:28.027291 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:26:28.027716 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:28.027722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:28.027749 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:28.027910 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:28.028636 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:28.028640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:28.028640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:28.028646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:28.028692 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:28.028714 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:28.028808 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:28.028847 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:26:28.028892 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260929 03:26:28.029649 16623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364: No bootstrap required, opened a new log
I20260929 03:26:28.030082 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:28.032407 16623 raft_consensus.cc:348] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260929 03:26:28.032567 16623 raft_consensus.cc:374] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:28.032590 16623 raft_consensus.cc:729] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f057c6da08ce434f91e39eceae63f364, State: Initialized, Role: FOLLOWER
I20260929 03:26:28.032766 16623 consensus_queue.cc:260] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260929 03:26:28.032887 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:28.031725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:50389 (local address 127.0.0.1:45127)
0929 03:26:28.031782 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:28.031784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:28.031912 (+ 128us) server_negotiation.cc:407] Connection header received
0929 03:26:28.031968 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:28.031968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:28.031975 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:28.032046 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:28.032671 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:28.032673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:28.032673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:28.032677 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:28.032691 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:28.032695 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:28.032742 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:28.032795 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:26:28.032810 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:26:28.032927 16623 raft_consensus.cc:388] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:28.032981 16623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:28.033032 16623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:28.033766 16623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260929 03:26:28.033861 16623 leader_election.cc:302] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f057c6da08ce434f91e39eceae63f364; no voters:
I20260929 03:26:28.033998 16623 leader_election.cc:288] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:28.034070 16633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:28.034258 16623 sys_catalog.cc:564] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:28.034256 16633 raft_consensus.cc:686] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [term 1 LEADER]: Becoming Leader. State: Replica: f057c6da08ce434f91e39eceae63f364, State: Running, Role: LEADER
I20260929 03:26:28.034469 16633 consensus_queue.cc:237] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } }
I20260929 03:26:28.034971 16634 sys_catalog.cc:455] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f057c6da08ce434f91e39eceae63f364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260929 03:26:28.035068 16634 sys_catalog.cc:458] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:28.035267 16635 sys_catalog.cc:455] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f057c6da08ce434f91e39eceae63f364. Latest consensus state: current_term: 1 leader_uuid: "f057c6da08ce434f91e39eceae63f364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f057c6da08ce434f91e39eceae63f364" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45127 } } }
I20260929 03:26:28.035351 16635 sys_catalog.cc:458] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:28.035498 16642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:28.036018 16642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:28.037717 16642 catalog_manager.cc:1347] Generated new cluster ID: 678e37635aaa4fc58f35182f714d425c
I20260929 03:26:28.037768 16642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:28.051977 16642 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:28.052199 16642 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:28.058269 16642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f057c6da08ce434f91e39eceae63f364: Generated new TSK 0
I20260929 03:26:28.058481 16642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:28.063903 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16555
2026-09-29T03:26:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:28 dist-test-slave-fzwk krb5kdc[16654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:28 dist-test-slave-fzwk krb5kdc[16654](info): set up 2 sockets
Sep 29 03:26:28 dist-test-slave-fzwk krb5kdc[16654](info): commencing operation
krb5kdc: starting...
W20260929 03:26:29.087670 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:29.206060 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:29.287863 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:29.288129 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:29.288170 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:29.289376 16667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:29.289458 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:29.289484 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:29.289506 16667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:29.289535 16667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:29.291095 16667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:29.291424 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:29.291667 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:29.296662 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:29.296662 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:29.296675 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:29.296680 16667 server_base.cc:1034] running on GCE node
I20260929 03:26:29.297153 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:29.297401 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:29.298552 16667 hybrid_clock.cc:648] HybridClock initialized: now 1790652389298538 us; error 30 us; skew 500 ppm
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:29.301069 16667 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:29.301568 16667 webserver.cc:460] Webserver started at http://127.0.0.1:34761/ using document root <none> and password file <none>
I20260929 03:26:29.301813 16667 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:29.301862 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:29.302031 16667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:29.303117 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "5345d186c8b94da095e992bc3d6e0e5c"
format_stamp: "Formatted at 2026-09-29 03:26:29 on dist-test-slave-fzwk"
server_key: "631e170b33220bf2f035cbdb8efe82c8"
server_key_iv: "5554caeef4004c5eb9a7c5dea27ebef4"
server_key_version: "encryptionkey@0"
I20260929 03:26:29.303550 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "5345d186c8b94da095e992bc3d6e0e5c"
format_stamp: "Formatted at 2026-09-29 03:26:29 on dist-test-slave-fzwk"
server_key: "631e170b33220bf2f035cbdb8efe82c8"
server_key_iv: "5554caeef4004c5eb9a7c5dea27ebef4"
server_key_version: "encryptionkey@0"
I20260929 03:26:29.304874 16667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:29.305608 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:29.305809 16667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:29.305898 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "5345d186c8b94da095e992bc3d6e0e5c"
format_stamp: "Formatted at 2026-09-29 03:26:29 on dist-test-slave-fzwk"
server_key: "631e170b33220bf2f035cbdb8efe82c8"
server_key_iv: "5554caeef4004c5eb9a7c5dea27ebef4"
server_key_version: "encryptionkey@0"
I20260929 03:26:29.305955 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:29.326992 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:29.329066 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:29.329200 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:29.333331 16667 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46683
I20260929 03:26:29.333410 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46683 every 8 connection(s)
I20260929 03:26:29.333804 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:29.334777 16735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:29.337306 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: Bootstrap starting.
I20260929 03:26:29.337939 16735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:29.338243 16735 log.cc:824] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:29.339041 16735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: No bootstrap required, opened a new log
I20260929 03:26:29.340777 16735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } }
I20260929 03:26:29.340988 16735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:29.341039 16735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345d186c8b94da095e992bc3d6e0e5c, State: Initialized, Role: FOLLOWER
I20260929 03:26:29.341223 16735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } }
I20260929 03:26:29.341323 16735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:29.341368 16735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:29.341420 16735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:29.341595 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16667
I20260929 03:26:29.341711 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:29.341921 27921 external_mini_cluster.cc:1428] Setting key 49343d21190821d8da1fe1f1a4d4a8e2
I20260929 03:26:29.342182 16735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } }
I20260929 03:26:29.342314 16735 leader_election.cc:302] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5345d186c8b94da095e992bc3d6e0e5c; no voters:
I20260929 03:26:29.342572 16735 leader_election.cc:288] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:29.342736 16741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:29.342855 16741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [term 1 LEADER]: Becoming Leader. State: Replica: 5345d186c8b94da095e992bc3d6e0e5c, State: Running, Role: LEADER
I20260929 03:26:29.342871 16735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:29.343016 16741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } }
I20260929 03:26:29.343699 16739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5345d186c8b94da095e992bc3d6e0e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } } }
I20260929 03:26:29.343854 16739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:29.344517 16742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5345d186c8b94da095e992bc3d6e0e5c. Latest consensus state: current_term: 1 leader_uuid: "5345d186c8b94da095e992bc3d6e0e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5345d186c8b94da095e992bc3d6e0e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46683 } } }
I20260929 03:26:29.344717 16742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:29.344939 16757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:29.345014 16757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:29.345081 16750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:29.345274 16750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:29.347025 16750 catalog_manager.cc:1347] Generated new cluster ID: 8f648d2d0fab440699a6778c3542cfb7
I20260929 03:26:29.347088 16750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:26:29.355882 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:29.342451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:55085 (local address 127.0.0.1:46683)
0929 03:26:29.342642 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:29.342656 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:29.342697 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:26:29.342851 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:29.342860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:29.342934 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:29.344034 (+ 1100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:29.344185 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:29.345765 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:29.346858 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:29.347514 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:29.347657 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:29.350179 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:29.350184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:29.350185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:29.350198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:29.353803 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:29.354371 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:29.354376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:29.354376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:29.354529 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:29.354793 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:29.354794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:29.354795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:29.355195 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:29.355423 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:29.355516 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:26:29.355630 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260929 03:26:29.356859 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:29.367391 16750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:29.368153 16750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:29.370414 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:29.363571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:41581 (local address 127.0.0.1:46683)
0929 03:26:29.363621 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:29.363623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:29.363843 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:26:29.363925 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:29.363926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:29.363937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:29.363994 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:29.364582 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:29.365097 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:29.365851 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:29.366071 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:29.366158 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:29.367717 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:29.367719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:29.367720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:29.367727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:29.369194 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:29.369589 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:29.369591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:29.369591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:29.369648 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:29.369873 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:29.369874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:29.369874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:29.369982 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:29.370127 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:29.370244 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:26:29.370310 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:26:29.374636 16750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5345d186c8b94da095e992bc3d6e0e5c: Generated new TSK 0
I20260929 03:26:29.374840 16750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:29.377748 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16667
W20260929 03:26:29.382100 16652 connection.cc:535] client connection to 127.0.0.1:46683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46683): Connection reset by peer (error 104)
2026-09-29T03:26:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16767](info): set up 2 sockets
Sep 29 03:26:29 dist-test-slave-fzwk krb5kdc[16767](info): commencing operation
krb5kdc: starting...
W20260929 03:26:30.423857 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:30 dist-test-slave-fzwk krb5kdc[16767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:30.546588 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:30.628690 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:30.628919 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:30.628962 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:30.630227 16780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:30.630280 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:30.630294 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:30.630308 16780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:30.630364 16780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:30.631950 16780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:30.632257 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:30.632507 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:30.637480 16785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:30.637473 16788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:30.637473 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:30.637586 16780 server_base.cc:1034] running on GCE node
I20260929 03:26:30.637990 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:30.638303 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:30.639451 16780 hybrid_clock.cc:648] HybridClock initialized: now 1790652390639425 us; error 36 us; skew 500 ppm
Sep 29 03:26:30 dist-test-slave-fzwk krb5kdc[16767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:30.647454 16780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:30.647969 16780 webserver.cc:460] Webserver started at http://127.0.0.1:35381/ using document root <none> and password file <none>
I20260929 03:26:30.648168 16780 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:30.648211 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:30.648304 16780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:30.649348 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "31e3e6729bb64265aa675582e8623e7f"
format_stamp: "Formatted at 2026-09-29 03:26:30 on dist-test-slave-fzwk"
server_key: "7d5ea3b1c9605c8f709a5cfbde49a098"
server_key_iv: "94caf7946fa7295c0fcc6a4f744b4758"
server_key_version: "encryptionkey@0"
I20260929 03:26:30.649672 16780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "31e3e6729bb64265aa675582e8623e7f"
format_stamp: "Formatted at 2026-09-29 03:26:30 on dist-test-slave-fzwk"
server_key: "7d5ea3b1c9605c8f709a5cfbde49a098"
server_key_iv: "94caf7946fa7295c0fcc6a4f744b4758"
server_key_version: "encryptionkey@0"
I20260929 03:26:30.651021 16780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:30.651907 16795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:30.652136 16780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:26:30.652254 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "31e3e6729bb64265aa675582e8623e7f"
format_stamp: "Formatted at 2026-09-29 03:26:30 on dist-test-slave-fzwk"
server_key: "7d5ea3b1c9605c8f709a5cfbde49a098"
server_key_iv: "94caf7946fa7295c0fcc6a4f744b4758"
server_key_version: "encryptionkey@0"
I20260929 03:26:30.652333 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:30.675034 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:30.677032 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:30.677222 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:30.681540 16780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33931
I20260929 03:26:30.681579 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33931 every 8 connection(s)
I20260929 03:26:30.681962 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:30.682202 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16780
I20260929 03:26:30.682322 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:30.682523 27921 external_mini_cluster.cc:1428] Setting key 5774899be34a76a55ab076d1f4638ab2
I20260929 03:26:30.682967 16848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:30.685767 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f: Bootstrap starting.
I20260929 03:26:30.686388 16848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:30.686691 16848 log.cc:824] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:30.687397 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f: No bootstrap required, opened a new log
I20260929 03:26:30.687922 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:30.683121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:34057 (local address 127.0.0.1:33931)
0929 03:26:30.683343 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:30.683351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:30.683378 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:26:30.683814 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:30.683820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:30.683873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:30.684031 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:30.684213 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:30.685127 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:30.685950 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:30.686369 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:30.686490 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:30.687272 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:30.687276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:30.687277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:30.687476 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:30.687515 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:30.687529 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:30.687629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:30.687710 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:26:30.687739 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260929 03:26:30.689173 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:30.689733 16848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } }
I20260929 03:26:30.689898 16848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:30.689946 16848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e3e6729bb64265aa675582e8623e7f, State: Initialized, Role: FOLLOWER
I20260929 03:26:30.690075 16848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } }
I20260929 03:26:30.690179 16848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:30.690219 16848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:30.690271 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:30.691005 16848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } }
I20260929 03:26:30.691130 16848 leader_election.cc:302] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e3e6729bb64265aa675582e8623e7f; no voters:
I20260929 03:26:30.691345 16848 leader_election.cc:288] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:30.691397 16858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:30.691592 16858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [term 1 LEADER]: Becoming Leader. State: Replica: 31e3e6729bb64265aa675582e8623e7f, State: Running, Role: LEADER
I20260929 03:26:30.691632 16848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:30.691715 16858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } }
I20260929 03:26:30.692075 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e3e6729bb64265aa675582e8623e7f. Latest consensus state: current_term: 1 leader_uuid: "31e3e6729bb64265aa675582e8623e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } } }
I20260929 03:26:30.692164 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:30.692088 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e3e6729bb64265aa675582e8623e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e3e6729bb64265aa675582e8623e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33931 } } }
I20260929 03:26:30.692314 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:30.692593 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:30.693349 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:30.694334 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:30.690597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:43659 (local address 127.0.0.1:33931)
0929 03:26:30.690633 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:30.690634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:30.690756 (+ 122us) server_negotiation.cc:407] Connection header received
0929 03:26:30.690831 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:30.690832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:30.690843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:30.690900 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:30.691050 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:30.691792 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:30.692847 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:30.693144 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:30.693207 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:30.694057 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:30.694058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:30.694058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:30.694063 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:30.694079 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:30.694084 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:30.694159 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:30.694251 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:26:30.694262 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:26:30.695145 16866 catalog_manager.cc:1347] Generated new cluster ID: 418cf9d156f34fedbb5d1a9d539445bd
I20260929 03:26:30.695200 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:30.702997 16866 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:30.703635 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:30.706274 16866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e3e6729bb64265aa675582e8623e7f: Generated new TSK 0
I20260929 03:26:30.706477 16866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:30.712657 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16780
W20260929 03:26:30.716671 16765 connection.cc:535] client connection to 127.0.0.1:33931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33931): Connection reset by peer (error 104)
2026-09-29T03:26:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:30 dist-test-slave-fzwk krb5kdc[16879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:30 dist-test-slave-fzwk krb5kdc[16879](info): set up 2 sockets
Sep 29 03:26:30 dist-test-slave-fzwk krb5kdc[16879](info): commencing operation
krb5kdc: starting...
W20260929 03:26:31.743649 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:31 dist-test-slave-fzwk krb5kdc[16879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:26:31.862490 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:31.944406 16892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:31.944633 16892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:31.944657 16892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:31.945789 16892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:31.945827 16892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:31.945842 16892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:31.945855 16892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:31.945868 16892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:31.947356 16892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:31.947563 16892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:31.947745 16892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:31.952514 16900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:31.952589 16897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:31.952761 16898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:31.952629 16892 server_base.cc:1034] running on GCE node
I20260929 03:26:31.953016 16892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:31.953358 16892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:31.954527 16892 hybrid_clock.cc:648] HybridClock initialized: now 1790652391954518 us; error 38 us; skew 500 ppm
Sep 29 03:26:31 dist-test-slave-fzwk krb5kdc[16879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:31.957017 16892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:26:31.957506 16892 webserver.cc:460] Webserver started at http://127.0.0.1:35607/ using document root <none> and password file <none>
I20260929 03:26:31.957746 16892 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:31.957820 16892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:31.957973 16892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:31.959065 16892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "ac941fed99b34826a83c7a93cd788b3b"
format_stamp: "Formatted at 2026-09-29 03:26:31 on dist-test-slave-fzwk"
server_key: "504c0a1421467e3546685288de62fe0c"
server_key_iv: "45d5394a80b7b20fd2c52c3d32beba16"
server_key_version: "encryptionkey@0"
I20260929 03:26:31.959473 16892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "ac941fed99b34826a83c7a93cd788b3b"
format_stamp: "Formatted at 2026-09-29 03:26:31 on dist-test-slave-fzwk"
server_key: "504c0a1421467e3546685288de62fe0c"
server_key_iv: "45d5394a80b7b20fd2c52c3d32beba16"
server_key_version: "encryptionkey@0"
I20260929 03:26:31.960878 16892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:31.961659 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:31.961831 16892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:31.961943 16892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "ac941fed99b34826a83c7a93cd788b3b"
format_stamp: "Formatted at 2026-09-29 03:26:31 on dist-test-slave-fzwk"
server_key: "504c0a1421467e3546685288de62fe0c"
server_key_iv: "45d5394a80b7b20fd2c52c3d32beba16"
server_key_version: "encryptionkey@0"
I20260929 03:26:31.962024 16892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:31.965615 16892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:31.967686 16892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:31.967849 16892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:31.972041 16892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39007
I20260929 03:26:31.972088 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39007 every 8 connection(s)
I20260929 03:26:31.972440 16892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:31.973430 16960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:31.975845 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b: Bootstrap starting.
I20260929 03:26:31.976449 16960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:31.976748 16960 log.cc:824] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:31.977456 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b: No bootstrap required, opened a new log
I20260929 03:26:31.977516 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 16892
I20260929 03:26:31.977640 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:31.977833 27921 external_mini_cluster.cc:1428] Setting key 7a66203e0b6c541f6c4278a2f448d426
I20260929 03:26:31.979940 16960 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260929 03:26:31.980103 16960 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:31.980149 16960 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac941fed99b34826a83c7a93cd788b3b, State: Initialized, Role: FOLLOWER
I20260929 03:26:31.980284 16960 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260929 03:26:31.980383 16960 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:31.980420 16960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:31.980477 16960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:31.980847 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:31.978394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:36837 (local address 127.0.0.1:39007)
0929 03:26:31.978591 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:31.978600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:31.978636 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:26:31.979467 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:31.979478 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:31.979515 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:31.979685 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:31.980343 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:31.980348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:31.980349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:31.980354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:31.980401 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:31.980416 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:31.980486 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:31.980569 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:26:31.980622 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260929 03:26:31.981532 16960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260929 03:26:31.981657 16960 leader_election.cc:302] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac941fed99b34826a83c7a93cd788b3b; no voters:
I20260929 03:26:31.981824 16960 leader_election.cc:288] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:31.981861 16965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:31.981871 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:31.982012 16965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [term 1 LEADER]: Becoming Leader. State: Replica: ac941fed99b34826a83c7a93cd788b3b, State: Running, Role: LEADER
I20260929 03:26:31.982128 16960 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:31.982141 16965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } }
I20260929 03:26:31.982605 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac941fed99b34826a83c7a93cd788b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } } }
I20260929 03:26:31.982605 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac941fed99b34826a83c7a93cd788b3b. Latest consensus state: current_term: 1 leader_uuid: "ac941fed99b34826a83c7a93cd788b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac941fed99b34826a83c7a93cd788b3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39007 } } }
I20260929 03:26:31.982712 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:31.982718 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:31.983033 16972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:31.983601 16972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:31.985641 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:31.983693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:39211 (local address 127.0.0.1:39007)
0929 03:26:31.983726 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:31.983728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:31.984621 (+ 893us) server_negotiation.cc:407] Connection header received
0929 03:26:31.984654 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:31.984655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:31.984661 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:31.984724 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:31.985368 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:31.985369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:31.985370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:31.985374 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:31.985391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:31.985395 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:31.985455 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:31.985568 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:26:31.985584 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:26:31.985831 16972 catalog_manager.cc:1347] Generated new cluster ID: aac45ed88be947b196b6abe1d2a1a1cd
I20260929 03:26:31.985880 16972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:32.000802 16972 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:32.001019 16972 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:32.019593 16972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac941fed99b34826a83c7a93cd788b3b: Generated new TSK 0
I20260929 03:26:32.019832 16972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:32.027855 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 16892
2026-09-29T03:26:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:32 dist-test-slave-fzwk krb5kdc[16991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:32 dist-test-slave-fzwk krb5kdc[16991](info): set up 2 sockets
Sep 29 03:26:32 dist-test-slave-fzwk krb5kdc[16991](info): commencing operation
krb5kdc: starting...
W20260929 03:26:33.059619 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:33.177811 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42733
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:43057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:33.260932 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:33.261197 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:33.261236 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:33.262422 17004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:33.262496 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:33.262521 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:33.262542 17004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:33.262566 17004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:33.264124 17004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:43057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:42733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:33.264436 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:33.264671 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:33.269676 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:33.269680 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:33.269680 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:33.270095 17004 server_base.cc:1034] running on GCE node
I20260929 03:26:33.270361 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:33.270660 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:33.271852 17004 hybrid_clock.cc:648] HybridClock initialized: now 1790652393271828 us; error 39 us; skew 500 ppm
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[16991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:33.281620 17004 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:33.282173 17004 webserver.cc:460] Webserver started at http://127.27.68.126:39419/ using document root <none> and password file <none>
I20260929 03:26:33.282424 17004 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:33.282495 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:33.282641 17004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:33.283826 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "2b5fcac5260c4d7ca6e9179afb5973f4"
format_stamp: "Formatted at 2026-09-29 03:26:33 on dist-test-slave-fzwk"
server_key: "772e7a5d7d59d6952c6e884a3f2adbe3"
server_key_iv: "afdf6c192cf5c9f93e0cedcbdd2d1b60"
server_key_version: "encryptionkey@0"
I20260929 03:26:33.284210 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "2b5fcac5260c4d7ca6e9179afb5973f4"
format_stamp: "Formatted at 2026-09-29 03:26:33 on dist-test-slave-fzwk"
server_key: "772e7a5d7d59d6952c6e884a3f2adbe3"
server_key_iv: "afdf6c192cf5c9f93e0cedcbdd2d1b60"
server_key_version: "encryptionkey@0"
I20260929 03:26:33.285573 17004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:33.286300 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:33.286484 17004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:33.286592 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "2b5fcac5260c4d7ca6e9179afb5973f4"
format_stamp: "Formatted at 2026-09-29 03:26:33 on dist-test-slave-fzwk"
server_key: "772e7a5d7d59d6952c6e884a3f2adbe3"
server_key_iv: "afdf6c192cf5c9f93e0cedcbdd2d1b60"
server_key_version: "encryptionkey@0"
I20260929 03:26:33.286669 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:33.298722 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:33.300851 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:33.301033 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:33.305446 17004 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:42733
I20260929 03:26:33.305481 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:42733 every 8 connection(s)
I20260929 03:26:33.305873 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:33.306779 17072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:33.309216 17072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4: Bootstrap starting.
I20260929 03:26:33.309871 17072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:33.310168 17072 log.cc:824] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:33.310885 17072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4: No bootstrap required, opened a new log
I20260929 03:26:33.312570 17072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } }
I20260929 03:26:33.312721 17072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:33.312743 17072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5fcac5260c4d7ca6e9179afb5973f4, State: Initialized, Role: FOLLOWER
I20260929 03:26:33.312948 17072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } }
I20260929 03:26:33.313015 17072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:33.313027 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17004
I20260929 03:26:33.313072 17072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:33.313123 17072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:33.313184 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:33.313366 27921 external_mini_cluster.cc:1428] Setting key 5d0450775773fcbf0644a2601500f1c9
I20260929 03:26:33.314270 17072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } }
I20260929 03:26:33.314396 17072 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5fcac5260c4d7ca6e9179afb5973f4; no voters:
I20260929 03:26:33.314568 17072 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:33.314599 17077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:33.314780 17077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b5fcac5260c4d7ca6e9179afb5973f4, State: Running, Role: LEADER
I20260929 03:26:33.314838 17072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:33.314970 17077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } }
I20260929 03:26:33.315557 17078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } } }
I20260929 03:26:33.315690 17078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:33.315954 17086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:33.316656 17086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:33.317201 17079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5fcac5260c4d7ca6e9179afb5973f4. Latest consensus state: current_term: 1 leader_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5fcac5260c4d7ca6e9179afb5973f4" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 42733 } } }
I20260929 03:26:33.317299 17079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:33.318663 17086 catalog_manager.cc:1347] Generated new cluster ID: 7a2efc4cee2f4dfea9b718dd72d3b033
I20260929 03:26:33.318763 17086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[16991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:33.324414 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:33.313894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35603 (local address 127.27.68.126:42733)
0929 03:26:33.314099 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:33.314107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:33.314135 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:26:33.314241 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:33.314247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:33.314292 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:33.314444 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:33.315313 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:33.316810 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:33.317908 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:33.318442 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:33.318606 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:33.321083 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:33.321087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:33.321088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:33.321107 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:33.322791 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:33.323197 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:33.323202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:33.323203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:33.323362 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:33.323614 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:33.323615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:33.323615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:33.323858 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:33.324064 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:33.324151 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:26:33.324253 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260929 03:26:33.325335 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:33.337160 17086 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:33.337896 17086 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:33.338459 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:33.326708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43463 (local address 127.27.68.126:42733)
0929 03:26:33.326751 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:33.326753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:33.331437 (+ 4684us) server_negotiation.cc:407] Connection header received
0929 03:26:33.331517 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:33.331518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:33.331530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:33.331607 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:33.332184 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:33.332693 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:33.333414 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:33.333680 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:33.333742 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:33.335108 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:33.335110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:33.335110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:33.335117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:33.337030 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:33.337487 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:33.337489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:33.337489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:33.337554 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:33.337792 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:33.337794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:33.337794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:33.337922 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:33.338059 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:33.338263 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 03:26:33.338360 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:33.346503 17086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5fcac5260c4d7ca6e9179afb5973f4: Generated new TSK 0
I20260929 03:26:33.346726 17086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:33.349895 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17004
W20260929 03:26:33.354444 16989 connection.cc:535] client connection to 127.27.68.126:42733 recv error: Network error: failed to read from TLS socket (remote: 127.27.68.126:42733): Connection reset by peer (error 104)
2026-09-29T03:26:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[17103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[17103](info): set up 2 sockets
Sep 29 03:26:33 dist-test-slave-fzwk krb5kdc[17103](info): commencing operation
krb5kdc: starting...
W20260929 03:26:34.391666 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:34 dist-test-slave-fzwk krb5kdc[17103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:34.510520 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35569
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:34.595522 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:34.595794 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:34.595839 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:34.597098 17116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:34.597178 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:34.597203 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:34.597223 17116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:34.597252 17116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:34.598848 17116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:34.599154 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:34.599422 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:34.604202 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:34.604260 17121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:34.604251 17124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:34.604420 17116 server_base.cc:1034] running on GCE node
I20260929 03:26:34.604693 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:34.605028 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:34.606210 17116 hybrid_clock.cc:648] HybridClock initialized: now 1790652394606198 us; error 34 us; skew 500 ppm
Sep 29 03:26:34 dist-test-slave-fzwk krb5kdc[17103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652394, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:34.608870 17116 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:34.609269 17116 webserver.cc:460] Webserver started at http://127.27.68.126:44129/ using document root <none> and password file <none>
I20260929 03:26:34.609468 17116 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:34.609508 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:34.609606 17116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:34.610630 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "2a5b822224d5469a86fdd10167a04014"
format_stamp: "Formatted at 2026-09-29 03:26:34 on dist-test-slave-fzwk"
server_key: "e6f44b8ecb851baff4e348aac2831220"
server_key_iv: "492b120ba2e79c11e3c4d9497bb81958"
server_key_version: "encryptionkey@0"
I20260929 03:26:34.610942 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "2a5b822224d5469a86fdd10167a04014"
format_stamp: "Formatted at 2026-09-29 03:26:34 on dist-test-slave-fzwk"
server_key: "e6f44b8ecb851baff4e348aac2831220"
server_key_iv: "492b120ba2e79c11e3c4d9497bb81958"
server_key_version: "encryptionkey@0"
I20260929 03:26:34.612339 17116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:34.613056 17131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:34.613209 17116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:34.613296 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "2a5b822224d5469a86fdd10167a04014"
format_stamp: "Formatted at 2026-09-29 03:26:34 on dist-test-slave-fzwk"
server_key: "e6f44b8ecb851baff4e348aac2831220"
server_key_iv: "492b120ba2e79c11e3c4d9497bb81958"
server_key_version: "encryptionkey@0"
I20260929 03:26:34.613355 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:34.626231 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:34.628192 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:34.628311 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:34.632259 17116 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35569
I20260929 03:26:34.632309 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35569 every 8 connection(s)
I20260929 03:26:34.632664 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:34.633545 17184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:34.635430 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17116
I20260929 03:26:34.635555 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:34.635746 27921 external_mini_cluster.cc:1428] Setting key ccde61a4e1af3185dec96280e8a9380a
I20260929 03:26:34.636145 17184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014: Bootstrap starting.
I20260929 03:26:34.636878 17184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:34.637172 17184 log.cc:824] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:34.637912 17184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014: No bootstrap required, opened a new log
I20260929 03:26:34.640128 17184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } }
I20260929 03:26:34.640332 17184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:34.640376 17184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a5b822224d5469a86fdd10167a04014, State: Initialized, Role: FOLLOWER
I20260929 03:26:34.640525 17184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } }
I20260929 03:26:34.640625 17184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:34.640671 17184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:34.640744 17184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:34.641324 17184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } }
I20260929 03:26:34.641446 17184 leader_election.cc:302] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a5b822224d5469a86fdd10167a04014; no voters:
I20260929 03:26:34.641623 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:34.636285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59949 (local address 127.27.68.126:35569)
0929 03:26:34.636484 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:34.636492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:34.636518 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:26:34.637084 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:34.637092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:34.637144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:34.637298 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:34.637558 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:34.638553 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:34.639726 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:34.640124 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:34.640204 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:34.640974 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:34.640978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:34.640978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:34.641162 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:34.641191 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:34.641202 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:34.641288 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:34.641371 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:26:34.641401 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260929 03:26:34.641647 17184 leader_election.cc:288] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:34.641892 17184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:34.641675 17189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:34.642181 17189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [term 1 LEADER]: Becoming Leader. State: Replica: 2a5b822224d5469a86fdd10167a04014, State: Running, Role: LEADER
I20260929 03:26:34.642336 17189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } }
I20260929 03:26:34.642602 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:34.642707 17189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a5b822224d5469a86fdd10167a04014. Latest consensus state: current_term: 1 leader_uuid: "2a5b822224d5469a86fdd10167a04014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } } }
I20260929 03:26:34.642802 17189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:34.643460 17199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:34.643801 17190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a5b822224d5469a86fdd10167a04014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a5b822224d5469a86fdd10167a04014" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35569 } } }
I20260929 03:26:34.643921 17190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014 [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:34.644215 17199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:34.645776 17199 catalog_manager.cc:1347] Generated new cluster ID: 468611d3b76c48608c0aee43994e296e
I20260929 03:26:34.645831 17199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:34.651149 17199 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:34.652140 17199 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:34.654793 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:34.644242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56243 (local address 127.27.68.126:35569)
0929 03:26:34.644285 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:34.644287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:34.651038 (+ 6751us) server_negotiation.cc:407] Connection header received
0929 03:26:34.651181 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:34.651182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:34.651196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:34.651300 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:34.651506 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:34.652239 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:34.653361 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:34.653685 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:34.653748 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:34.654530 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:34.654532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:34.654532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:34.654538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:34.654554 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:34.654559 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:34.654613 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:34.654705 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:26:34.654715 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:26:34.656389 17199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a5b822224d5469a86fdd10167a04014: Generated new TSK 0
I20260929 03:26:34.656589 17199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:34.661550 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17116
W20260929 03:26:34.666011 17101 connection.cc:535] client connection to 127.27.68.126:35569 recv error: Network error: failed to read from TLS socket (remote: 127.27.68.126:35569): Connection reset by peer (error 104)
2026-09-29T03:26:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:34 dist-test-slave-fzwk krb5kdc[17215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:34 dist-test-slave-fzwk krb5kdc[17215](info): set up 2 sockets
Sep 29 03:26:34 dist-test-slave-fzwk krb5kdc[17215](info): commencing operation
krb5kdc: starting...
W20260929 03:26:35.703578 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:35 dist-test-slave-fzwk krb5kdc[17215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:35.822291 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:45973
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:35.904680 17228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:35.904947 17228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:35.904990 17228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:35.906225 17228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:35.906301 17228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:35.906325 17228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:35.906347 17228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:35.906376 17228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:35.908001 17228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:45973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:35.908314 17228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:35.908551 17228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:35.913479 17234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:35.913437 17233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:35.913437 17236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:35.914110 17228 server_base.cc:1034] running on GCE node
I20260929 03:26:35.914333 17228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:35.914613 17228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:35.915809 17228 hybrid_clock.cc:648] HybridClock initialized: now 1790652395915795 us; error 35 us; skew 500 ppm
Sep 29 03:26:35 dist-test-slave-fzwk krb5kdc[17215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652395, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:35.924440 17228 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:35.924955 17228 webserver.cc:460] Webserver started at http://127.27.68.126:35511/ using document root <none> and password file <none>
I20260929 03:26:35.925204 17228 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:35.925274 17228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:35.925429 17228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:35.926791 17228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "0fdad169afa84e62a05c0b1d047804cc"
format_stamp: "Formatted at 2026-09-29 03:26:35 on dist-test-slave-fzwk"
server_key: "fd578b801da817cfc8a0fc15c7fa36c7"
server_key_iv: "08db1885213cfa311c174cdb2a49b85a"
server_key_version: "encryptionkey@0"
I20260929 03:26:35.927268 17228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "0fdad169afa84e62a05c0b1d047804cc"
format_stamp: "Formatted at 2026-09-29 03:26:35 on dist-test-slave-fzwk"
server_key: "fd578b801da817cfc8a0fc15c7fa36c7"
server_key_iv: "08db1885213cfa311c174cdb2a49b85a"
server_key_version: "encryptionkey@0"
I20260929 03:26:35.928709 17228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:26:35.929476 17243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:35.929670 17228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:35.929776 17228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "0fdad169afa84e62a05c0b1d047804cc"
format_stamp: "Formatted at 2026-09-29 03:26:35 on dist-test-slave-fzwk"
server_key: "fd578b801da817cfc8a0fc15c7fa36c7"
server_key_iv: "08db1885213cfa311c174cdb2a49b85a"
server_key_version: "encryptionkey@0"
I20260929 03:26:35.929863 17228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:35.933378 17228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:35.935834 17228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:35.936000 17228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:35.940131 17228 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:45973
I20260929 03:26:35.940186 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:45973 every 8 connection(s)
I20260929 03:26:35.940608 17228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:35.941632 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:35.944162 17296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc: Bootstrap starting.
I20260929 03:26:35.944804 17296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:35.945102 17296 log.cc:824] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:35.945851 17296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc: No bootstrap required, opened a new log
I20260929 03:26:35.947328 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17228
I20260929 03:26:35.947439 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:35.947645 27921 external_mini_cluster.cc:1428] Setting key d77da1aa37823de5e28ad63fedd01ced
I20260929 03:26:35.947638 17296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } }
I20260929 03:26:35.947857 17296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:35.947901 17296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fdad169afa84e62a05c0b1d047804cc, State: Initialized, Role: FOLLOWER
I20260929 03:26:35.948069 17296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } }
I20260929 03:26:35.948176 17296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:35.948251 17296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:35.948315 17296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:35.949576 17296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } }
I20260929 03:26:35.949697 17296 leader_election.cc:302] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fdad169afa84e62a05c0b1d047804cc; no voters:
I20260929 03:26:35.949869 17296 leader_election.cc:288] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:35.950047 17302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:35.950104 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:35.948214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47629 (local address 127.27.68.126:45973)
0929 03:26:35.948518 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:35.948529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:35.948564 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:26:35.948695 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:35.948703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:35.948751 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:35.948917 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:35.949598 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:35.949603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:35.949604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:35.949608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:35.949661 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:35.949676 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:35.949756 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:35.949830 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:26:35.949879 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260929 03:26:35.950125 17296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:35.951189 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:35.950189 17302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [term 1 LEADER]: Becoming Leader. State: Replica: 0fdad169afa84e62a05c0b1d047804cc, State: Running, Role: LEADER
I20260929 03:26:35.951361 17302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } }
I20260929 03:26:35.952888 17303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fdad169afa84e62a05c0b1d047804cc. Latest consensus state: current_term: 1 leader_uuid: "0fdad169afa84e62a05c0b1d047804cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } } }
I20260929 03:26:35.952999 17303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:35.952986 17301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fdad169afa84e62a05c0b1d047804cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fdad169afa84e62a05c0b1d047804cc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 45973 } } }
I20260929 03:26:35.953091 17301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:35.953430 17322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:35.953982 17322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:35.954545 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:35.952752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55389 (local address 127.27.68.126:45973)
0929 03:26:35.952785 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:35.952788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:35.953448 (+ 660us) server_negotiation.cc:407] Connection header received
0929 03:26:35.953527 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:35.953528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:35.953534 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:35.953598 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:35.954304 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:35.954306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:35.954306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:35.954311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:35.954327 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:35.954331 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:35.954388 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:35.954449 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:26:35.954463 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:26:35.955891 17322 catalog_manager.cc:1347] Generated new cluster ID: 1e1bd2519fd94e74a356cd1b38c5bb3d
I20260929 03:26:35.955950 17322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:35.973291 17322 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:35.973505 17322 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:35.978682 17322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fdad169afa84e62a05c0b1d047804cc: Generated new TSK 0
I20260929 03:26:35.978902 17322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:35.985082 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17228
2026-09-29T03:26:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:36 dist-test-slave-fzwk krb5kdc[17327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:36 dist-test-slave-fzwk krb5kdc[17327](info): set up 2 sockets
Sep 29 03:26:36 dist-test-slave-fzwk krb5kdc[17327](info): commencing operation
krb5kdc: starting...
W20260929 03:26:37.011682 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:37.130569 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46273
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:46273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:37.212884 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:37.213115 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:37.213140 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:37.214262 17340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:37.214298 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:37.214313 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:37.214324 17340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:37.214337 17340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:37.215772 17340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:46273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:46273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:37.215988 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:37.216168 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:37.220935 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:37.220960 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:37.221139 17340 server_base.cc:1034] running on GCE node
W20260929 03:26:37.220937 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:37.221422 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:37.221742 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:37.222906 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790652397222894 us; error 39 us; skew 500 ppm
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652397, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:37.225409 17340 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:37.225857 17340 webserver.cc:460] Webserver started at http://127.27.68.126:46425/ using document root <none> and password file <none>
I20260929 03:26:37.226111 17340 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:37.226181 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:37.226320 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:37.227473 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "e6738523a39b409ebd55f550c5ef86fb"
format_stamp: "Formatted at 2026-09-29 03:26:37 on dist-test-slave-fzwk"
server_key: "6f9fcaa1fed3247f614f96996ceae102"
server_key_iv: "5e41c11de38c80ee51f92e9bd03f3e08"
server_key_version: "encryptionkey@0"
I20260929 03:26:37.227856 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "e6738523a39b409ebd55f550c5ef86fb"
format_stamp: "Formatted at 2026-09-29 03:26:37 on dist-test-slave-fzwk"
server_key: "6f9fcaa1fed3247f614f96996ceae102"
server_key_iv: "5e41c11de38c80ee51f92e9bd03f3e08"
server_key_version: "encryptionkey@0"
I20260929 03:26:37.229177 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:37.229915 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:37.230089 17340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:37.230192 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "e6738523a39b409ebd55f550c5ef86fb"
format_stamp: "Formatted at 2026-09-29 03:26:37 on dist-test-slave-fzwk"
server_key: "6f9fcaa1fed3247f614f96996ceae102"
server_key_iv: "5e41c11de38c80ee51f92e9bd03f3e08"
server_key_version: "encryptionkey@0"
I20260929 03:26:37.230268 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:37.245441 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:37.247421 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:37.247609 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:37.251652 17340 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:46273
I20260929 03:26:37.251688 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:46273 every 8 connection(s)
I20260929 03:26:37.252089 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:37.252993 17408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:37.255367 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb: Bootstrap starting.
I20260929 03:26:37.255446 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17340
I20260929 03:26:37.255584 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:37.255781 27921 external_mini_cluster.cc:1428] Setting key 45b5e08bd4f90e554b65bcb346c0cb28
I20260929 03:26:37.256078 17408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:37.256462 17408 log.cc:824] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:37.257258 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb: No bootstrap required, opened a new log
I20260929 03:26:37.259698 17408 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } }
I20260929 03:26:37.259878 17408 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:37.259905 17408 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6738523a39b409ebd55f550c5ef86fb, State: Initialized, Role: FOLLOWER
I20260929 03:26:37.260057 17408 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } }
I20260929 03:26:37.260128 17408 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:37.260150 17408 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:37.260217 17408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:37.260991 17408 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } }
I20260929 03:26:37.261109 17408 leader_election.cc:302] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6738523a39b409ebd55f550c5ef86fb; no voters:
I20260929 03:26:37.261288 17408 leader_election.cc:288] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:37.261348 17413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:37.261551 17413 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [term 1 LEADER]: Becoming Leader. State: Replica: e6738523a39b409ebd55f550c5ef86fb, State: Running, Role: LEADER
I20260929 03:26:37.261678 17408 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:37.261747 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } }
I20260929 03:26:37.262171 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6738523a39b409ebd55f550c5ef86fb. Latest consensus state: current_term: 1 leader_uuid: "e6738523a39b409ebd55f550c5ef86fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } } }
I20260929 03:26:37.262303 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:37.262581 17421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:37.263175 17421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:37.263622 17414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6738523a39b409ebd55f550c5ef86fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6738523a39b409ebd55f550c5ef86fb" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 46273 } } }
I20260929 03:26:37.263711 17414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb [sys.catalog]: This master's current role is: LEADER
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:37.264969 17421 catalog_manager.cc:1347] Generated new cluster ID: 904a4400bf234f639c01ce1f2acea96f
I20260929 03:26:37.265027 17421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:37.267776 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:37.256341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48005 (local address 127.27.68.126:46273)
0929 03:26:37.256547 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:37.256557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:37.256591 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:26:37.257227 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:37.257236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:37.257299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:37.257456 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:37.258039 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:37.259593 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:37.260591 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:37.261263 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:37.261384 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:37.264422 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:37.264427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:37.264427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:37.264441 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:37.266184 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:37.266572 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:37.266575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:37.266576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:37.266672 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:37.266881 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:37.266882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:37.266883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:37.267136 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:37.267295 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:37.267449 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:26:37.267597 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260929 03:26:37.268704 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:37.271418 17421 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:37.271922 17421 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:37.277081 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:37.270306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39467 (local address 127.27.68.126:46273)
0929 03:26:37.270367 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:37.270369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:37.270423 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:26:37.270498 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:37.270499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:37.270507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:37.270548 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:37.271347 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:37.271914 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:37.272892 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:37.273088 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:37.273140 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:37.274429 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:37.274430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:37.274430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:37.274436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:37.275891 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:37.276279 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:37.276281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:37.276281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:37.276348 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:37.276541 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:37.276542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:37.276542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:37.276681 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:37.276785 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:37.276895 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:26:37.276989 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:26:37.280108 17421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6738523a39b409ebd55f550c5ef86fb: Generated new TSK 0
I20260929 03:26:37.280303 17421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:37.288432 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17340
2026-09-29T03:26:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17439](info): set up 2 sockets
Sep 29 03:26:37 dist-test-slave-fzwk krb5kdc[17439](info): commencing operation
krb5kdc: starting...
W20260929 03:26:38.323822 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:38 dist-test-slave-fzwk krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:38.447011 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41441
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:41441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:38.532994 17452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:38.533236 17452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:38.533277 17452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:38.534493 17452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:38.534561 17452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:38.534590 17452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:38.534619 17452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:38.534649 17452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:38.536224 17452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:41441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:41441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:38.536528 17452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:38.536758 17452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:38.541733 17457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:38.541741 17460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:38.541996 17458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:38.541975 17452 server_base.cc:1034] running on GCE node
I20260929 03:26:38.542237 17452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:38.542541 17452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:38.543705 17452 hybrid_clock.cc:648] HybridClock initialized: now 1790652398543695 us; error 46 us; skew 500 ppm
Sep 29 03:26:38 dist-test-slave-fzwk krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652398, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:38.546083 17452 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:38.546582 17452 webserver.cc:460] Webserver started at http://127.27.68.126:33319/ using document root <none> and password file <none>
I20260929 03:26:38.546855 17452 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:38.546934 17452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:38.547077 17452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:38.548239 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "e59be21d09984f2695db9fd41a94e86c"
format_stamp: "Formatted at 2026-09-29 03:26:38 on dist-test-slave-fzwk"
server_key: "08b20dfa18cdb673a46472036ab9e60d"
server_key_iv: "c868ddea5d602910af2c74a3ec82465a"
server_key_version: "encryptionkey@0"
I20260929 03:26:38.548614 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "e59be21d09984f2695db9fd41a94e86c"
format_stamp: "Formatted at 2026-09-29 03:26:38 on dist-test-slave-fzwk"
server_key: "08b20dfa18cdb673a46472036ab9e60d"
server_key_iv: "c868ddea5d602910af2c74a3ec82465a"
server_key_version: "encryptionkey@0"
I20260929 03:26:38.549953 17452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:38.550671 17467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:38.550843 17452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:38.550944 17452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "e59be21d09984f2695db9fd41a94e86c"
format_stamp: "Formatted at 2026-09-29 03:26:38 on dist-test-slave-fzwk"
server_key: "08b20dfa18cdb673a46472036ab9e60d"
server_key_iv: "c868ddea5d602910af2c74a3ec82465a"
server_key_version: "encryptionkey@0"
I20260929 03:26:38.551029 17452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:38.558320 17452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:38.560246 17452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:38.560416 17452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:38.564630 17452 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:41441
I20260929 03:26:38.564662 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:41441 every 8 connection(s)
I20260929 03:26:38.565039 17452 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:38.565919 17520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:38.568336 17520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c: Bootstrap starting.
I20260929 03:26:38.568903 17520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:38.569145 17520 log.cc:824] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:38.569830 17520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c: No bootstrap required, opened a new log
I20260929 03:26:38.571457 17520 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } }
I20260929 03:26:38.571606 17520 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:38.571630 17520 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59be21d09984f2695db9fd41a94e86c, State: Initialized, Role: FOLLOWER
I20260929 03:26:38.571760 17520 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } }
I20260929 03:26:38.571839 17520 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:38.571861 17520 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:38.571894 17520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:38.572289 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17452
I20260929 03:26:38.572413 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:38.572640 27921 external_mini_cluster.cc:1428] Setting key 229827d032e79c598e4e58294093cc27
I20260929 03:26:38.573038 17520 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } }
I20260929 03:26:38.573169 17520 leader_election.cc:302] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59be21d09984f2695db9fd41a94e86c; no voters:
I20260929 03:26:38.573344 17520 leader_election.cc:288] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:38.573366 17525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:38.573570 17525 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [term 1 LEADER]: Becoming Leader. State: Replica: e59be21d09984f2695db9fd41a94e86c, State: Running, Role: LEADER
I20260929 03:26:38.573731 17520 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:38.573730 17525 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } }
I20260929 03:26:38.574189 17526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59be21d09984f2695db9fd41a94e86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } } }
I20260929 03:26:38.574278 17526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:38.574478 17527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59be21d09984f2695db9fd41a94e86c. Latest consensus state: current_term: 1 leader_uuid: "e59be21d09984f2695db9fd41a94e86c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59be21d09984f2695db9fd41a94e86c" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 41441 } } }
I20260929 03:26:38.574579 17527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:38.575145 17532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:38.575845 17532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:38.577486 17532 catalog_manager.cc:1347] Generated new cluster ID: ff02e649767a4306af31764b096d87e2
I20260929 03:26:38.577575 17532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:38.578684 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:38.573151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59877 (local address 127.27.68.126:41441)
0929 03:26:38.573382 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:38.573392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:38.573432 (+ 40us) server_negotiation.cc:407] Connection header received
0929 03:26:38.573571 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:38.573579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:38.573644 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:38.574046 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:38.574150 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:38.575666 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:38.576517 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:38.577127 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:38.577238 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:38.578016 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:38.578020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:38.578020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:38.578207 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:38.578238 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:38.578250 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:38.578324 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:38.578401 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:26:38.578467 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260929 03:26:38.579586 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:38.583794 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:38.580944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59005 (local address 127.27.68.126:41441)
0929 03:26:38.580983 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:38.580985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:38.581106 (+ 121us) server_negotiation.cc:407] Connection header received
0929 03:26:38.581178 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:38.581179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:38.581189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:38.581233 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:38.581382 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:38.581848 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:38.582557 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:38.582770 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:38.582850 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:38.583550 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:38.583551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:38.583552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:38.583556 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:38.583570 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:38.583575 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:38.583623 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:38.583685 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:26:38.583709 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:26:38.589944 17532 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:38.590677 17532 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:38.601470 17532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59be21d09984f2695db9fd41a94e86c: Generated new TSK 0
I20260929 03:26:38.601703 17532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:38.604974 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17452
2026-09-29T03:26:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:38 dist-test-slave-fzwk krb5kdc[17551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:38 dist-test-slave-fzwk krb5kdc[17551](info): set up 2 sockets
Sep 29 03:26:38 dist-test-slave-fzwk krb5kdc[17551](info): commencing operation
krb5kdc: starting...
W20260929 03:26:39.632123 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:39 dist-test-slave-fzwk krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:39.750449 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35773
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:40377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:35773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:39.831909 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:39.832188 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:39.832228 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:39.833431 17564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:39.833504 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:39.833528 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:39.833567 17564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:39.833596 17564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:39.835136 17564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:40377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:35773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:35773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:39.835440 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:39.835685 17564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:39.840576 17569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:39.840610 17570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:39.840572 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:39.840814 17564 server_base.cc:1034] running on GCE node
I20260929 03:26:39.841133 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:39.841466 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:39.842634 17564 hybrid_clock.cc:648] HybridClock initialized: now 1790652399842624 us; error 39 us; skew 500 ppm
Sep 29 03:26:39 dist-test-slave-fzwk krb5kdc[17551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652399, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:39.845240 17564 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:39.845736 17564 webserver.cc:460] Webserver started at http://127.27.68.126:40701/ using document root <none> and password file <none>
I20260929 03:26:39.845965 17564 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:39.846014 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:39.846155 17564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:39.847355 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "65f96b3c2a6642a59990a8874c9d5f5d"
format_stamp: "Formatted at 2026-09-29 03:26:39 on dist-test-slave-fzwk"
server_key: "0793045e9b2e111b04b413723b3b2eae"
server_key_iv: "3b6aea1cf68d58f4570d60281a55f338"
server_key_version: "encryptionkey@0"
I20260929 03:26:39.847719 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "65f96b3c2a6642a59990a8874c9d5f5d"
format_stamp: "Formatted at 2026-09-29 03:26:39 on dist-test-slave-fzwk"
server_key: "0793045e9b2e111b04b413723b3b2eae"
server_key_iv: "3b6aea1cf68d58f4570d60281a55f338"
server_key_version: "encryptionkey@0"
I20260929 03:26:39.849057 17564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:39.849819 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:39.850000 17564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:39.850092 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "65f96b3c2a6642a59990a8874c9d5f5d"
format_stamp: "Formatted at 2026-09-29 03:26:39 on dist-test-slave-fzwk"
server_key: "0793045e9b2e111b04b413723b3b2eae"
server_key_iv: "3b6aea1cf68d58f4570d60281a55f338"
server_key_version: "encryptionkey@0"
I20260929 03:26:39.850150 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:39.853534 17564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:39.855624 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:39.855736 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:39.859700 17564 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:35773
I20260929 03:26:39.859761 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:35773 every 8 connection(s)
I20260929 03:26:39.860121 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:39.861130 17632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:39.863487 17632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: Bootstrap starting.
I20260929 03:26:39.864043 17632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:39.864295 17632 log.cc:824] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:39.864967 17632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: No bootstrap required, opened a new log
I20260929 03:26:39.865252 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17564
I20260929 03:26:39.865357 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:39.865556 27921 external_mini_cluster.cc:1428] Setting key 2db92e74b1043b312e9e395811110484
I20260929 03:26:39.867090 17632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } }
I20260929 03:26:39.867334 17632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:26:39.867419 17632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f96b3c2a6642a59990a8874c9d5f5d, State: Initialized, Role: FOLLOWER
I20260929 03:26:39.867602 17632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } }
I20260929 03:26:39.867724 17632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:26:39.867764 17632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:26:39.867816 17632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:39.868559 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:39.866091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58617 (local address 127.27.68.126:35773)
0929 03:26:39.866263 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:39.866271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:39.866297 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:26:39.867061 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:39.867067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:39.867095 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:39.867275 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:39.868086 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:39.868090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:39.868090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:39.868094 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:39.868134 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:39.868145 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:39.868201 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:39.868293 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:26:39.868345 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260929 03:26:39.868980 17632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } }
I20260929 03:26:39.869113 17632 leader_election.cc:302] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f96b3c2a6642a59990a8874c9d5f5d; no voters:
I20260929 03:26:39.869293 17632 leader_election.cc:288] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:26:39.869336 17637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:39.869530 17637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [term 1 LEADER]: Becoming Leader. State: Replica: 65f96b3c2a6642a59990a8874c9d5f5d, State: Running, Role: LEADER
I20260929 03:26:39.869565 17632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:39.869655 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:39.869673 17637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } }
I20260929 03:26:39.869993 17638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } } }
I20260929 03:26:39.870100 17638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:39.869984 17637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f96b3c2a6642a59990a8874c9d5f5d. Latest consensus state: current_term: 1 leader_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f96b3c2a6642a59990a8874c9d5f5d" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 35773 } } }
I20260929 03:26:39.870280 17637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:39.871533 17656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:39.871591 17656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:39.871758 17645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:39.872000 17645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:39.872778 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:39.871487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44145 (local address 127.27.68.126:35773)
0929 03:26:39.871528 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:39.871529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:39.871765 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:26:39.871816 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:39.871817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:39.871824 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:39.871882 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:39.872553 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:39.872554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:39.872555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:26:39.872559 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:39.872576 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:26:39.872581 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:39.872645 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:39.872704 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:26:39.872719 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:26:39.873620 17645 catalog_manager.cc:1347] Generated new cluster ID: cd926cb8d233499782bbf92169bca80b
I20260929 03:26:39.873675 17645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:39.888159 17645 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:39.888356 17645 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:39.892097 17645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f96b3c2a6642a59990a8874c9d5f5d: Generated new TSK 0
I20260929 03:26:39.892305 17645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:39.894491 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17564
2026-09-29T03:26:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1289 ms)
[----------] 18 tests from AuthTokenIssuingTest (23722 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:39 dist-test-slave-fzwk krb5kdc[17663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:39 dist-test-slave-fzwk krb5kdc[17663](info): set up 2 sockets
Sep 29 03:26:39 dist-test-slave-fzwk krb5kdc[17663](info): commencing operation
krb5kdc: starting...
W20260929 03:26:40.927737 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:41.046096 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:39283
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:41.127739 17676 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:41.128002 17676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:41.128033 17676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:41.128079 17676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:41.129297 17676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:41.129365 17676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:41.129390 17676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:41.129410 17676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:41.129441 17676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:41.131029 17676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:41.131345 17676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:41.131587 17676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:41.136507 17682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:41.136629 17676 server_base.cc:1034] running on GCE node
W20260929 03:26:41.136482 17681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:41.136720 17684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:41.136963 17676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:41.137238 17676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:41.138386 17676 hybrid_clock.cc:648] HybridClock initialized: now 1790652401138366 us; error 37 us; skew 500 ppm
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:41.144434 17676 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:41.144979 17676 webserver.cc:460] Webserver started at http://127.27.68.126:45409/ using document root <none> and password file <none>
I20260929 03:26:41.145224 17676 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:41.145274 17676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:41.145431 17676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:41.146538 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "19d825b95d79436fa1708a35a45474fc"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "2be92eece8fc8c002a2ad533cd257b96"
server_key_iv: "c718300eb9ec133973c9bde74083c744"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.146924 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "19d825b95d79436fa1708a35a45474fc"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "2be92eece8fc8c002a2ad533cd257b96"
server_key_iv: "c718300eb9ec133973c9bde74083c744"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.148427 17676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260929 03:26:41.149222 17691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:41.149430 17676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:26:41.149531 17676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "19d825b95d79436fa1708a35a45474fc"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "2be92eece8fc8c002a2ad533cd257b96"
server_key_iv: "c718300eb9ec133973c9bde74083c744"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.149670 17676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:41.172458 17676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:41.174414 17676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:41.174590 17676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:41.178875 17676 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:39283
I20260929 03:26:41.178912 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:39283 every 8 connection(s)
I20260929 03:26:41.179304 17676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:41.180305 17744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:41.181252 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17676
I20260929 03:26:41.181347 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:41.181540 27921 external_mini_cluster.cc:1428] Setting key 01c304c6c2d6a62a0000ff19e70f51bc
I20260929 03:26:41.182104 17744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.68.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.125@KRBTEST.COM" created.
I20260929 03:26:41.193037 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.182887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.27.68.126:39283)
0929 03:26:41.183123 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.183133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.183235 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:26:41.183811 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.183814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.183837 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.183944 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.184494 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.185885 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.186760 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.187328 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.187446 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.189782 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.189791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.189792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.189799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.191401 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.191781 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.191784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.191784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.191897 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.192172 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.192173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.192174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.192436 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.192518 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.192759 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.192859 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"spinlock_wait_cycles":9472,"thread_start_us":119,"threads_started":1}
I20260929 03:26:41.193166 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.182879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:39283 (local address 127.0.0.1:48158)
0929 03:26:41.183085 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.183126 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.183651 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.183961 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.183963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.184229 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.184389 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.184393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.185991 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.185992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.186677 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.186679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.187429 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.187430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.187485 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.188066 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.188074 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.189696 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.191472 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.191479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.191481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.191682 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.191982 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.191983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.191983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.192107 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.192513 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.192520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.192610 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.192718 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.192969 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":60,"thread_start_us":110,"threads_started":1}
I20260929 03:26:41.193962 17744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.194345 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.194229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55658)
0929 03:26:41.194273 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.194318 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:26:41.194500 17693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
I20260929 03:26:41.194950 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.194863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55668)
0929 03:26:41.194909 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.194934 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:26:41.195063 17744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:38417: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
I20260929 03:26:41.207723 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:38417
--webserver_interface=127.27.68.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:26:41.244285 17744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } attempt: 1
I20260929 03:26:41.244783 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.244658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55676)
0929 03:26:41.244713 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.244752 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:26:41.245143 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.245040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55690)
0929 03:26:41.245087 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.245113 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:26:41.245231 17744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:38417: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
W20260929 03:26:41.289688 17751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:41.289994 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:41.290035 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:41.290093 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:41.291388 17751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:41.291463 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:41.291486 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:41.291507 17751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:41.291533 17751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:41.293174 17751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.68.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:41.293493 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:41.293731 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:41.298768 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:41.298768 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:41.298810 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:41.298978 17751 server_base.cc:1034] running on GCE node
I20260929 03:26:41.299342 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:41.299651 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:41.300818 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790652401300792 us; error 36 us; skew 500 ppm
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:41.303388 17751 init.cc:377] Logged in from keytab as kudu/127.27.68.125@KRBTEST.COM (short username kudu)
I20260929 03:26:41.303867 17751 webserver.cc:460] Webserver started at http://127.27.68.125:39961/ using document root <none> and password file <none>
I20260929 03:26:41.304103 17751 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:41.304171 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:41.304322 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:41.305430 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data/instance:
uuid: "3be7704e091c4f709d4f8595dae90abd"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "cae3791d6d4ac6a99a46af777fe03852"
server_key_iv: "1fe3b27915d04373926cf077f68329be"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.305810 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal/instance:
uuid: "3be7704e091c4f709d4f8595dae90abd"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "cae3791d6d4ac6a99a46af777fe03852"
server_key_iv: "1fe3b27915d04373926cf077f68329be"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.307165 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:41.307945 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:41.308120 17751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:41.308220 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "3be7704e091c4f709d4f8595dae90abd"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "cae3791d6d4ac6a99a46af777fe03852"
server_key_iv: "1fe3b27915d04373926cf077f68329be"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.308300 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:41.313369 17744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } attempt: 2
I20260929 03:26:41.313819 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.313689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55696)
0929 03:26:41.313747 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.313788 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:26:41.314206 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.314111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:55710)
0929 03:26:41.314154 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.314182 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:26:41.314323 17744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:38417: Network error: Client connection negotiation failed: client connection to 127.27.68.125:38417: connect: Connection refused (error 111)
I20260929 03:26:41.324754 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:41.326659 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:41.326824 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:41.330889 17751 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.125:38417
I20260929 03:26:41.330941 17818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.125:38417 every 8 connection(s)
I20260929 03:26:41.331393 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
I20260929 03:26:41.332427 17819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:41.332492 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17751
I20260929 03:26:41.332621 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-1/wal/instance
I20260929 03:26:41.332808 27921 external_mini_cluster.cc:1428] Setting key e0c953374760ec83b06c855d55ca1278
I20260929 03:26:41.334334 17819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.68.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.124@KRBTEST.COM" created.
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:41.346879 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.335190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.27.68.126:39283)
0929 03:26:41.335271 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.335273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.335490 (+ 217us) server_negotiation.cc:407] Connection header received
0929 03:26:41.335999 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.336001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.336016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.336059 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.336759 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.337249 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.338236 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.338451 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.338534 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.343538 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.343540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.343540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.343550 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.345131 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.345734 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.345735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.345736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.345811 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.346161 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.346162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.346162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.346299 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.346357 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.346734 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.346808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:26:41.347203 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.335173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:39283 (local address 127.0.0.1:37542)
0929 03:26:41.335371 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.335408 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.335891 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.336126 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.336129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.336450 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.336670 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.336675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.337351 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.337352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.338148 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.338150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.338561 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.338562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.338689 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.339198 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.339207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.343429 (+ 4222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.345234 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.345242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.345249 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.345652 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.345908 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.345909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.345909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.346076 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.346376 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.346384 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.346590 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.346717 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.346910 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260929 03:26:41.347819 17819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:41.358606 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.348138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:43056)
0929 03:26:41.348190 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.348201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.348237 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.348499 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.348500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.348686 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.348760 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.348762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.349614 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.349615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.350495 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.350498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.350980 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.350981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.351068 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.351501 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.351508 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.355004 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.357223 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.357224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.357224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.357392 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.357634 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.357635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.357636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.357690 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.358214 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.358215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.358263 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.358346 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.358472 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:26:41.358825 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.348193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.27.68.125:38417)
0929 03:26:41.348351 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.348356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.348375 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:26:41.348400 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.348402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.348414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.348475 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.348835 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.349524 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.350591 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.350902 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.350958 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.355128 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.355160 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.355160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.355168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.357127 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.357510 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.357513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.357513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.357570 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.357755 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.357756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.357756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.357953 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.358049 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.358416 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.358595 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
I20260929 03:26:41.361760 17819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.362190 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.362033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52188)
0929 03:26:41.362093 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.362153 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:26:41.362208 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:37409
--webserver_interface=127.27.68.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:41.362342 17770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
I20260929 03:26:41.362937 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.362818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52202)
0929 03:26:41.362869 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.362903 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:26:41.363071 17819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:37409: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
I20260929 03:26:41.405465 17744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } attempt: 3
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:41.412266 17819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } attempt: 1
I20260929 03:26:41.412710 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.412565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52218)
0929 03:26:41.412632 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.412672 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:26:41.413074 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.412985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52226)
0929 03:26:41.413016 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.413039 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 03:26:41.413213 17819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:37409: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
I20260929 03:26:41.414026 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.405854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:43068)
0929 03:26:41.405962 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.405974 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.406018 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.406304 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.406305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.406510 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.406586 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.406588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.407570 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.407572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.408342 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.408347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.408958 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.408959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.409050 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.409649 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.409654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.410966 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.412982 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.412984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.412984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.413222 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.413439 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.413440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.413441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.413506 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.413781 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.413782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.413825 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.413873 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.413961 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 03:26:41.414068 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.406120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.27.68.125:38417)
0929 03:26:41.406162 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.406164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.406173 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:41.406200 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.406200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.406211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.406254 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.406678 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.407453 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.408464 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.408818 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.408888 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.411074 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.411075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.411076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.411084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.412904 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.413313 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.413314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.413314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.413376 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.413573 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.413574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.413575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.413717 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.413793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.413898 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.413984 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:26:41.414426 17744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.414776 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.414675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52238)
0929 03:26:41.414720 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.414747 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:26:41.415096 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.415038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52246)
0929 03:26:41.415063 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.415082 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260929 03:26:41.415191 17744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:37409: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
I20260929 03:26:41.446458 17744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } attempt: 1
I20260929 03:26:41.447045 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.446881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52260)
0929 03:26:41.446956 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.447008 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:26:41.447458 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.447351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52266)
0929 03:26:41.447398 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.447432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:26:41.447570 17744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:37409: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
W20260929 03:26:41.449335 17826 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:41.449536 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:41.449571 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:41.449591 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:41.450867 17826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:41.450910 17826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:41.450925 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:41.450955 17826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:41.450987 17826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:41.452570 17826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:39283,127.27.68.125:38417,127.27.68.124:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.68.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:41.452811 17826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:41.453004 17826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:41.457976 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:41.457984 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:41.457984 17834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:41.458019 17826 server_base.cc:1034] running on GCE node
I20260929 03:26:41.458384 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:41.458690 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:41.459823 17826 hybrid_clock.cc:648] HybridClock initialized: now 1790652401459813 us; error 30 us; skew 500 ppm
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:41.462244 17826 init.cc:377] Logged in from keytab as kudu/127.27.68.124@KRBTEST.COM (short username kudu)
I20260929 03:26:41.462661 17826 webserver.cc:460] Webserver started at http://127.27.68.124:34633/ using document root <none> and password file <none>
I20260929 03:26:41.462844 17826 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:41.462885 17826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:41.462975 17826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:41.464074 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data/instance:
uuid: "8031c0a354fe4cf48796ad205af0663e"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "1b251fe7486fe58c9d91ea03ad76f667"
server_key_iv: "2a21b5ea771a6bba30a746394eb9ccd9"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.464372 17826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal/instance:
uuid: "8031c0a354fe4cf48796ad205af0663e"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "1b251fe7486fe58c9d91ea03ad76f667"
server_key_iv: "2a21b5ea771a6bba30a746394eb9ccd9"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.465655 17826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:26:41.466321 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:41.466468 17826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:26:41.466560 17826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "8031c0a354fe4cf48796ad205af0663e"
format_stamp: "Formatted at 2026-09-29 03:26:41 on dist-test-slave-fzwk"
server_key: "1b251fe7486fe58c9d91ea03ad76f667"
server_key_iv: "2a21b5ea771a6bba30a746394eb9ccd9"
server_key_version: "encryptionkey@0"
I20260929 03:26:41.466617 17826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:41.476694 17826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:41.478506 17826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:41.478621 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:41.481357 17819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } attempt: 2
I20260929 03:26:41.481832 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.481665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52270)
0929 03:26:41.481734 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.481782 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:26:41.482203 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.482102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52274)
0929 03:26:41.482140 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.482172 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 03:26:41.482292 17819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:37409: Network error: Client connection negotiation failed: client connection to 127.27.68.124:37409: connect: Connection refused (error 111)
I20260929 03:26:41.482736 17826 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.124:37409
I20260929 03:26:41.482765 17893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.124:37409 every 8 connection(s)
I20260929 03:26:41.483122 17826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
I20260929 03:26:41.484128 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:41.485674 17894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.487507 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 17826
I20260929 03:26:41.487630 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652132259900-27921-0/minicluster-data/master-2/wal/instance
I20260929 03:26:41.487849 27921 external_mini_cluster.cc:1428] Setting key 310f35cd6245cfa6b7bbc029875cdc4d
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:41.495920 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.486344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.27.68.126:39283)
0929 03:26:41.486408 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.486411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.486614 (+ 203us) server_negotiation.cc:407] Connection header received
0929 03:26:41.487090 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.487091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.487101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.487155 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.488074 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.488768 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.489937 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.490232 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.490283 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.492769 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.492771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.492771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.492779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.494258 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.494833 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.494834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.494835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.494895 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.495265 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.495266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.495266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.495395 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.495452 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.495778 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.495851 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:26:41.496206 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.486322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:39283 (local address 127.0.0.1:37544)
0929 03:26:41.486483 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.486523 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.487007 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.487252 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.487257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.487666 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.487965 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.487972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.488858 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.488859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.489852 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.489855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.490325 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.490327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.490470 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.490997 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.491009 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.492669 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.494372 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.494380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.494382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.494760 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.494985 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.494986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.494987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.495155 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.495463 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.495468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.495632 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.495750 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.495941 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260929 03:26:41.496893 17894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.497241 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.488232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.27.68.124:37409)
0929 03:26:41.488406 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.488410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.488466 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:26:41.488502 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.488504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.488523 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.488604 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.489300 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.490077 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.491047 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.491397 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.491453 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.494080 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.494090 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.494090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.494098 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.495986 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.496372 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.496374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.496375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.496428 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.496658 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.496658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.496659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.496811 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.496955 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.497048 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.497127 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:41.504391 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.497180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:38417 (local address 127.0.0.1:43084)
0929 03:26:41.497289 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.497300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.497370 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.497670 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.497671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.497810 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.497871 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.497872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.498785 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.498786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.499363 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.499365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.499714 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.499715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.499789 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.500182 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.500186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.501676 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.503413 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.503414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.503415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.503580 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.503790 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.503791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.503792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.503859 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.504153 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.504154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.504191 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.504245 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.504321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 03:26:41.504492 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.497234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.27.68.125:38417)
0929 03:26:41.497281 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.497283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.497343 (+ 60us) server_negotiation.cc:407] Connection header received
0929 03:26:41.497434 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.497435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.497447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.497494 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.497949 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.498713 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.499440 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.499645 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.499700 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.501775 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.501776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.501777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.501784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.503325 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.503657 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.503659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.503659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.503717 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.503922 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.503923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.503923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.504073 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.504150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.504292 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.504374 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:26:41.504813 17894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:41.505872 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.497994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.27.68.126:39283)
0929 03:26:41.498039 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.498040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.498082 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:26:41.498149 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.498150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.498157 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.498199 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.498721 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.499155 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.500016 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.500292 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.500344 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.502899 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.502900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.502901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.502908 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.504614 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.505011 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.505013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.505013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.505083 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.505353 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.505354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.505354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.505470 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.505657 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.505694 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.505769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:41.511660 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.505091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52292)
0929 03:26:41.505141 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.505151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.505188 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.505289 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.505290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.505416 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.505443 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.505444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.506174 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.506175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.506792 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.506794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.507167 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.507167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.507263 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.507657 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.507662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.508928 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.510500 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.510501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.510502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.510717 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.510929 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.510930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.510931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.511000 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.511289 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.511290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.511336 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.511407 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.511512 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:26:41.511734 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.505155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.27.68.124:37409)
0929 03:26:41.505199 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.505200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.505209 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:41.505230 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.505230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.505241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.505289 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.505515 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.506096 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.506878 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.507091 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.507167 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.509011 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.509012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.509012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.509017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.510423 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.510788 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.510789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.510789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.510857 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.511057 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.511058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.511058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.511202 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.511299 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.511439 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.511555 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":29}
I20260929 03:26:41.513530 17894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: Bootstrap starting.
I20260929 03:26:41.514315 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.506503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.27.68.125:38417)
0929 03:26:41.506546 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.506548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.506575 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:26:41.506594 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.506594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.506603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.506646 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.507098 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.507634 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.508432 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.508634 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.508729 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.511437 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.511438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.511438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.511445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.513141 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.513476 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.513477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.513477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.513539 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.513760 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.513760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.513761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.513893 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.514092 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.514142 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.514227 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:41.514580 17894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:41.514822 17894 log.cc:824] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:41.515727 17894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: No bootstrap required, opened a new log
I20260929 03:26:41.515964 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:41.517349 17894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.517500 17894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8031c0a354fe4cf48796ad205af0663e, State: Initialized, Role: FOLLOWER
I20260929 03:26:41.517598 17894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.517038 17718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "19d825b95d79436fa1708a35a45474fc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37550
I20260929 03:26:41.517951 17894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:41.517903 17900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.518030 17900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:26:41.519713 17911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:41.519788 17911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:41.522670 17744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } attempt: 2
I20260929 03:26:41.520040 17826 master_runner.cc:186] Error getting master registration for 127.27.68.126:39283: OK, instance_id { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" instance_seqno: 1790652401177632 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:26:41.522857 17826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:41.530997 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.523018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.27.68.124:37409)
0929 03:26:41.523054 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.523057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.523065 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:26:41.523086 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.523087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.523096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.523140 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.523510 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.524255 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.525246 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.525576 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.525648 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.527923 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.527925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.527925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.527937 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.529653 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.530088 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.530089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.530090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.530156 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.530356 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.530357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.530357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.530543 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.530619 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.530814 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.530899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:26:41.530998 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.522928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52306)
0929 03:26:41.522969 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.522989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.523033 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.523153 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.523154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.523376 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.523427 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.523429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.524352 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.524353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.525163 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.525165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.525666 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.525667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.525720 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.526164 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.526169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.527824 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.529730 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.529731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.529732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.530009 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.530227 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.530228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.530228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.530292 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.530633 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.530635 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.530692 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.530778 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.530890 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:26:41.532756 17744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: Bootstrap starting.
I20260929 03:26:41.533545 17744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:41.533838 17744 log.cc:824] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:41.534607 17744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: No bootstrap required, opened a new log
I20260929 03:26:41.535743 17744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.535892 17744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d825b95d79436fa1708a35a45474fc, State: Initialized, Role: FOLLOWER
I20260929 03:26:41.536010 17744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.536294 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.536365 17744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:41.536381 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:41.536509 17718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:26:41.536589 17718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:41.537425 17718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.537676 17718 leader_election.cc:288] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [CANDIDATE]: Term 1 election: Requested vote from peers 3be7704e091c4f709d4f8595dae90abd (127.27.68.125:38417), 8031c0a354fe4cf48796ad205af0663e (127.27.68.124:37409)
I20260929 03:26:41.538069 17868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d825b95d79436fa1708a35a45474fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8031c0a354fe4cf48796ad205af0663e"
I20260929 03:26:41.538239 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:41.539331 17868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d825b95d79436fa1708a35a45474fc in term 1.
I20260929 03:26:41.539585 17695 leader_election.cc:302] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d825b95d79436fa1708a35a45474fc, 8031c0a354fe4cf48796ad205af0663e; no voters:
I20260929 03:26:41.539723 17914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:41.539875 17914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [term 1 LEADER]: Becoming Leader. State: Replica: 19d825b95d79436fa1708a35a45474fc, State: Running, Role: LEADER
I20260929 03:26:41.539974 17914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.540627 17932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d825b95d79436fa1708a35a45474fc. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.540730 17932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: This master's current role is: LEADER
W20260929 03:26:41.541522 17929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:41.541568 17929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:41.542110 17934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:41.542261 17934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:41.540342 17676 master_runner.cc:186] Error getting master registration for 127.27.68.125:38417: OK, instance_id { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" instance_seqno: 1790652401329740 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:26:41.542593 17676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:26:41.549192 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.539047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.27.68.125:38417)
0929 03:26:41.539092 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.539095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.539996 (+ 901us) server_negotiation.cc:407] Connection header received
0929 03:26:41.540271 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.540272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.540286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.540322 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.541306 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.542014 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.543364 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.544196 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.544247 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.545975 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.545977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.545977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.545984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.547679 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.548316 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.548317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.548318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.548377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.548585 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.548586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.548586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.548718 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.548787 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.549009 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.549090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:41.551944 17868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [term 1 FOLLOWER]: Refusing update from remote peer 19d825b95d79436fa1708a35a45474fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:41.552302 17914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:41.554088 17900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d825b95d79436fa1708a35a45474fc. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.554196 17900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:41.558192 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.543251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.27.68.126:39283)
0929 03:26:41.543291 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.543293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.550018 (+ 6725us) server_negotiation.cc:407] Connection header received
0929 03:26:41.550122 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.550123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.550134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.550166 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.550998 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.551767 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.553006 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.553301 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.553339 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.555129 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.555131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.555131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.555138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.556976 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.557349 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.557350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.557350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.557402 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.557590 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.557591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.557591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.557717 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.557787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.558009 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.558093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:41.558372 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.539575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.27.68.124:37409)
0929 03:26:41.539631 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.539633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.547768 (+ 8135us) server_negotiation.cc:407] Connection header received
0929 03:26:41.547796 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.547797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.547809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.547859 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.548884 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.549632 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.551795 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.552057 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.552113 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.554657 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.554659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.554659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.554666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.556447 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.557123 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.557140 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.557141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.557200 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.557565 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.557566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.557566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.557718 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.557801 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.558177 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.558258 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:26:41.564352 17914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.564464 17914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:41.565109 17934 catalog_manager.cc:1347] Generated new cluster ID: 4cac49e323a84c479bd66f9bd0b2cc4d
I20260929 03:26:41.565160 17934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:41.565120 17900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.565201 17900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:41.565570 17936 mvcc.cc:205] Tried to move back new op lower bound from 7334512236751568896 to 7334512236708409344. Current Snapshot: MvccSnapshot[applied={T|T < 7334512236751568896 or (T in {7334512236751568896})}]
I20260929 03:26:41.565722 17938 mvcc.cc:205] Tried to move back new op lower bound from 7334512236751568896 to 7334512236708409344. Current Snapshot: MvccSnapshot[applied={T|T < 7334512236751568896}]
I20260929 03:26:41.573382 17819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } attempt: 3
I20260929 03:26:41.578105 17934 catalog_manager.cc:1370] Generated new certificate authority record
Sep 29 03:26:41 dist-test-slave-fzwk krb5kdc[17663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652401, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:41.578876 17934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:41.582881 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.573662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:37409 (local address 127.0.0.1:52322)
0929 03:26:41.573718 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:41.573729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:41.573770 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:41.573911 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:41.573912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:41.574124 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:41.574174 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.574176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.575056 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.575058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.575920 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:41.575923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.576389 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.576390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:41.576446 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.577035 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:41.577041 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:41.579656 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:41.581643 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.581644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.581645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.581886 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.582147 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:41.582148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:41.582148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:41.582219 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:41.582543 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:41.582544 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:41.582606 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 03:26:41.582667 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:26:41.582776 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:26:41.583484 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:41.573726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.27.68.124:37409)
0929 03:26:41.573767 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.573769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.573777 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:26:41.573844 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.573845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.573856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.573908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.574239 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.574987 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.576012 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:41.576323 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:41.576384 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:41.579754 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:41.579756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:41.579756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:41.579764 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:41.581518 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.581988 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.581989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.581990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.582064 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:41.582297 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:41.582298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:41.582298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:41.582441 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:41.582498 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:41.583303 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 03:26:41.583395 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:41.583787 17934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc: Generated new TSK 0
I20260929 03:26:41.584139 17934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:26:41.585460 17819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: Bootstrap starting.
I20260929 03:26:41.586210 17819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:41.586513 17819 log.cc:824] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:41.600033 17819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: No bootstrap required, opened a new log
I20260929 03:26:41.602198 17819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.602371 17819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be7704e091c4f709d4f8595dae90abd, State: Initialized, Role: FOLLOWER
I20260929 03:26:41.602487 17819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } }
I20260929 03:26:41.602841 17819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:41.602921 17793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d825b95d79436fa1708a35a45474fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3be7704e091c4f709d4f8595dae90abd"
I20260929 03:26:41.603034 17793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:26:41.603264 17898 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.601836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.27.68.124:37409)
0929 03:26:41.601876 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.601878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.602694 (+ 816us) server_negotiation.cc:407] Connection header received
0929 03:26:41.602791 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.602792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.602803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.602832 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.603161 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52334: BlockingRecv error: recv got EOF from 127.0.0.1:52334 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:26:41.603615 17748 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.601559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.27.68.126:39283)
0929 03:26:41.601602 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.601604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.602146 (+ 542us) server_negotiation.cc:407] Connection header received
0929 03:26:41.602246 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.602247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.602258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.602287 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.603539 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37560: BlockingRecv error: recv got EOF from 127.0.0.1:37560 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:26:41.603963 17793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d825b95d79436fa1708a35a45474fc in term 1.
I20260929 03:26:41.604051 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.604156 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:41.604179 17792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [term 1 FOLLOWER]: Refusing update from remote peer 19d825b95d79436fa1708a35a45474fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:41.604739 17932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:26:41.604805 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.601465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.27.68.125:38417)
0929 03:26:41.601509 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.601511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.602032 (+ 521us) server_negotiation.cc:407] Connection header received
0929 03:26:41.602198 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.602199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.602223 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.602252 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.604753 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43112: BlockingRecv error: recv got EOF from 127.0.0.1:43112 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:26:41.606632 17963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:41.606729 17963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:41.606787 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d825b95d79436fa1708a35a45474fc. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.606881 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:41.606972 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:41.607100 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:26:41.609306 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.608787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.27.68.125:38417)
0929 03:26:41.608825 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.608827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.608836 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:41.608863 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.608864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.608879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.608945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.609260 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43126: BlockingRecv error: recv got EOF from 127.0.0.1:43126 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:26:41.612533 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.611689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.27.68.125:38417)
0929 03:26:41.611731 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.611733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.611954 (+ 221us) server_negotiation.cc:407] Connection header received
0929 03:26:41.612049 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.612050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.612086 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.612115 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.612483 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43136: BlockingRecv error: recv got EOF from 127.0.0.1:43136 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:26:41.616755 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.615931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.27.68.125:38417)
0929 03:26:41.615974 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.615976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.616208 (+ 232us) server_negotiation.cc:407] Connection header received
0929 03:26:41.616302 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.616303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.616313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.616355 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.616705 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43148: BlockingRecv error: recv got EOF from 127.0.0.1:43148 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 03:26:41.622900 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.622079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.27.68.125:38417)
0929 03:26:41.622119 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.622121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.622358 (+ 237us) server_negotiation.cc:407] Connection header received
0929 03:26:41.622464 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.622465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.622476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.622504 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.622842 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43154: BlockingRecv error: recv got EOF from 127.0.0.1:43154 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:26:41.633219 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.632354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.27.68.125:38417)
0929 03:26:41.632406 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.632408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.632664 (+ 256us) server_negotiation.cc:407] Connection header received
0929 03:26:41.632764 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.632766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.632777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.632806 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.633163 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43170: BlockingRecv error: recv got EOF from 127.0.0.1:43170 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:26:41.651810 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.650770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.27.68.125:38417)
0929 03:26:41.650814 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.650816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.651107 (+ 291us) server_negotiation.cc:407] Connection header received
0929 03:26:41.651256 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.651257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.651271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.651304 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.651747 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43184: BlockingRecv error: recv got EOF from 127.0.0.1:43184 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:26:41.686409 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.685439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.27.68.125:38417)
0929 03:26:41.685487 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.685489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.685769 (+ 280us) server_negotiation.cc:407] Connection header received
0929 03:26:41.685877 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.685878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.685891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.685925 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.686330 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43186: BlockingRecv error: recv got EOF from 127.0.0.1:43186 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:26:41.753002 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.752199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.27.68.125:38417)
0929 03:26:41.752256 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.752259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.752355 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:26:41.752441 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.752442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.752456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.752524 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.752929 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43198: BlockingRecv error: recv got EOF from 127.0.0.1:43198 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 03:26:41.882884 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:41.882231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.27.68.125:38417)
0929 03:26:41.882320 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:41.882322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:41.882381 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:26:41.882434 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:41.882435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:41.882446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:41.882511 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:41.882816 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43202: BlockingRecv error: recv got EOF from 127.0.0.1:43202 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 03:26:42.140767 17824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:42.140148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.27.68.125:38417)
0929 03:26:42.140219 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.140221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.140233 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:26:42.140299 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.140300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.140311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.140376 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:42.140694 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43206: BlockingRecv error: recv got EOF from 127.0.0.1:43206 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:26:42.215085 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19d825b95d79436fa1708a35a45474fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d825b95d79436fa1708a35a45474fc" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 39283 } } peers { permanent_uuid: "3be7704e091c4f709d4f8595dae90abd" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 38417 } } peers { permanent_uuid: "8031c0a354fe4cf48796ad205af0663e" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 37409 } } }
I20260929 03:26:42.215258 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d825b95d79436fa1708a35a45474fc [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:42.520335 17911 catalog_manager.cc:1259] Loaded cluster ID: 4cac49e323a84c479bd66f9bd0b2cc4d
I20260929 03:26:42.520431 17911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: loading cluster ID for follower catalog manager: success
I20260929 03:26:42.521807 17911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: acquiring CA information for follower catalog manager: success
I20260929 03:26:42.522094 17911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8031c0a354fe4cf48796ad205af0663e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:26:42.607180 17963 catalog_manager.cc:1259] Loaded cluster ID: 4cac49e323a84c479bd66f9bd0b2cc4d
I20260929 03:26:42.607306 17963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: loading cluster ID for follower catalog manager: success
I20260929 03:26:42.608110 17963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: acquiring CA information for follower catalog manager: success
I20260929 03:26:42.608417 17963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3be7704e091c4f709d4f8595dae90abd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:26:42.656643 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:42.654029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.27.68.125:38417)
0929 03:26:42.654155 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.654157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.654171 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:26:42.654256 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.654257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.654313 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.654380 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:42.654510 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.655018 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.655826 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.656032 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.656094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:42.656252 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:42.656434 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:42.656501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:42.656540 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 03:26:42.656569 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260929 03:26:42.661140 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:42.658516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.27.68.124:37409)
0929 03:26:42.658641 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.658643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.658653 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:26:42.658719 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.658720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.658772 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.658860 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:42.658956 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.659535 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.660279 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.660500 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.660579 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:42.660749 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:42.660868 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:42.660950 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:42.661008 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:26:42.661033 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260929 03:26:42.665892 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:42.662819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.27.68.125:38417)
0929 03:26:42.662872 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.662875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.663069 (+ 194us) server_negotiation.cc:407] Connection header received
0929 03:26:42.663111 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.663112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.663127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.663189 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:42.663345 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.664132 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665084 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.665374 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665436 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:42.665615 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:42.665650 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:42.665712 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:42.665770 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:26:42.665793 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:26:42.666033 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:42.662816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.27.68.126:39283)
0929 03:26:42.662928 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.662931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.663133 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:26:42.663202 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.663202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.663275 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.663356 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:42.663543 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.664301 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665014 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.665315 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:42.665576 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:42.665686 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:42.665745 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:42.665912 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:26:42.665938 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260929 03:26:42.666287 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:42.662812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.27.68.124:37409)
0929 03:26:42.662854 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:42.662856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:42.663116 (+ 260us) server_negotiation.cc:407] Connection header received
0929 03:26:42.663235 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:42.663236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:42.663248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:42.663275 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:42.663474 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.663985 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665473 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:42.665760 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:42.665821 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:42.665983 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:42.666020 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:42.666074 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:42.666172 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:26:42.666198 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:42.668202 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17676
I20260929 03:26:42.673893 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17751
I20260929 03:26:42.679075 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 17826
2026-09-29T03:26:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2785 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:26:42 dist-test-slave-fzwk krb5kdc[18046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:26:42 dist-test-slave-fzwk krb5kdc[18046](info): set up 2 sockets
Sep 29 03:26:42 dist-test-slave-fzwk krb5kdc[18046](info): commencing operation
krb5kdc: starting...
W20260929 03:26:43.727829 27921 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/test-user.keytab.
Sep 29 03:26:43 dist-test-slave-fzwk krb5kdc[18046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:26:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.68.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
Entry for principal kudu/127.27.68.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab.
I20260929 03:26:43.846158 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34177
--webserver_interface=127.27.68.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:26:43.928467 18059 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:43.928711 18059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:43.928730 18059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:43.928781 18059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:43.929984 18059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:43.930029 18059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:43.930047 18059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:43.930060 18059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:43.930114 18059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:43.931782 18059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.126.keytab
--principal=kudu/127.27.68.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.126:34177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.68.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:43.932071 18059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:43.932317 18059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:43.937251 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:43.937235 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:43.937235 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:43.937456 18059 server_base.cc:1034] running on GCE node
I20260929 03:26:43.937712 18059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:43.938017 18059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:43.939169 18059 hybrid_clock.cc:648] HybridClock initialized: now 1790652403939147 us; error 34 us; skew 500 ppm
Sep 29 03:26:43 dist-test-slave-fzwk krb5kdc[18046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:43.941725 18059 init.cc:377] Logged in from keytab as kudu/127.27.68.126@KRBTEST.COM (short username kudu)
I20260929 03:26:43.942150 18059 webserver.cc:460] Webserver started at http://127.27.68.126:43183/ using document root <none> and password file <none>
I20260929 03:26:43.942337 18059 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:43.942378 18059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:43.942480 18059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:43.943619 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data/instance:
uuid: "92c58cb9bb0b4c11ab9603e42f784f9b"
format_stamp: "Formatted at 2026-09-29 03:26:43 on dist-test-slave-fzwk"
server_key: "69bb50bb1370ea15219cd503f3e3856c"
server_key_iv: "8ff453b0778ce41bbf00a3f394785fe0"
server_key_version: "encryptionkey@0"
I20260929 03:26:43.943928 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance:
uuid: "92c58cb9bb0b4c11ab9603e42f784f9b"
format_stamp: "Formatted at 2026-09-29 03:26:43 on dist-test-slave-fzwk"
server_key: "69bb50bb1370ea15219cd503f3e3856c"
server_key_iv: "8ff453b0778ce41bbf00a3f394785fe0"
server_key_version: "encryptionkey@0"
I20260929 03:26:43.945220 18059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:26:43.945906 18074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:43.946089 18059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:26:43.946192 18059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal
uuid: "92c58cb9bb0b4c11ab9603e42f784f9b"
format_stamp: "Formatted at 2026-09-29 03:26:43 on dist-test-slave-fzwk"
server_key: "69bb50bb1370ea15219cd503f3e3856c"
server_key_iv: "8ff453b0778ce41bbf00a3f394785fe0"
server_key_version: "encryptionkey@0"
I20260929 03:26:43.946266 18059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:43.967831 18059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:43.969741 18059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:43.969918 18059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:43.973920 18059 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.126:34177
I20260929 03:26:43.973960 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.126:34177 every 8 connection(s)
I20260929 03:26:43.974330 18059 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/data/info.pb
I20260929 03:26:43.975402 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:43.977020 18127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:43.981114 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 18059
I20260929 03:26:43.981199 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-0/wal/instance
I20260929 03:26:43.981410 27921 external_mini_cluster.cc:1428] Setting key 43917a91395ac03f0bb6ff29d9c9af46
Sep 29 03:26:43 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:43.987339 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:43.977771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.27.68.126:34177)
0929 03:26:43.977957 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:43.977961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:43.978032 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:26:43.978565 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:43.978567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:43.978584 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:43.978662 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:43.979198 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:43.980072 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:43.980966 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:43.981338 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:43.981428 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:43.983824 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:43.983832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:43.983832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:43.983841 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:43.985559 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:43.986044 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:43.986047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:43.986047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:43.986127 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:43.986405 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:43.986406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:43.986406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:43.986633 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:43.986695 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:43.986949 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 03:26:43.987079 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260929 03:26:43.987358 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:43.977727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34177 (local address 127.0.0.1:58392)
0929 03:26:43.977896 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:43.977944 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:43.978436 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:43.978686 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:43.978689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:43.978942 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:43.979118 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:43.979122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:43.980156 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:43.980157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:43.980844 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:43.980845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:43.981441 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:43.981442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:43.981528 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:43.982059 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:43.982069 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:43.983732 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:43.985651 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:43.985660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:43.985663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:43.985960 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:43.986196 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:43.986197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:43.986198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:43.986331 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:43.986699 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:43.986707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:43.986801 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:26:43.986921 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 03:26:43.987152 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":47,"thread_start_us":76,"threads_started":1}
I20260929 03:26:43.988207 18127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:43.988595 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:43.988447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59386)
0929 03:26:43.988497 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:43.988562 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:26:43.988747 18078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
I20260929 03:26:43.989213 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:43.989097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59402)
0929 03:26:43.989148 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:43.989182 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:26:43.989331 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:44635: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.68.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
Entry for principal kudu/127.27.68.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab.
I20260929 03:26:44.006198 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:44635
--webserver_interface=127.27.68.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:26:44.038676 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } attempt: 1
I20260929 03:26:44.039302 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.039109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59418)
0929 03:26:44.039175 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.039264 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:26:44.039675 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.039596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59430)
0929 03:26:44.039634 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.039660 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:26:44.039777 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:44635: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
W20260929 03:26:44.089401 18134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:44.089673 18134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:44.089704 18134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:44.089764 18134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:44.090978 18134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:44.091044 18134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:44.091075 18134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:44.091101 18134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:44.091130 18134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:44.092739 18134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.125.keytab
--principal=kudu/127.27.68.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.125:44635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.68.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:44.093041 18134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:44.093292 18134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:44.098335 18139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:44.098351 18142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:44.098390 18134 server_base.cc:1034] running on GCE node
W20260929 03:26:44.098356 18140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:44.098786 18134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:44.099121 18134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:44.100296 18134 hybrid_clock.cc:648] HybridClock initialized: now 1790652404100283 us; error 32 us; skew 500 ppm
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:44.102911 18134 init.cc:377] Logged in from keytab as kudu/127.27.68.125@KRBTEST.COM (short username kudu)
I20260929 03:26:44.103477 18134 webserver.cc:460] Webserver started at http://127.27.68.125:34963/ using document root <none> and password file <none>
I20260929 03:26:44.103731 18134 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:44.103803 18134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:44.103947 18134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:44.105042 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data/instance:
uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "9b936c5e486917bd75c87b346b3554dd"
server_key_iv: "587c0dc36a8e50ffe975f220ed081d3f"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.105427 18134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal/instance:
uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "9b936c5e486917bd75c87b346b3554dd"
server_key_iv: "587c0dc36a8e50ffe975f220ed081d3f"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.106820 18134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:26:44.107650 18149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:44.107825 18134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:44.107931 18134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal
uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "9b936c5e486917bd75c87b346b3554dd"
server_key_iv: "587c0dc36a8e50ffe975f220ed081d3f"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.107918 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } attempt: 2
I20260929 03:26:44.108011 18134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:44.108325 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.108220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59434)
0929 03:26:44.108260 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.108297 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:26:44.108652 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.108576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59440)
0929 03:26:44.108615 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.108638 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:26:44.108753 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.125:44635: Network error: Client connection negotiation failed: client connection to 127.27.68.125:44635: connect: Connection refused (error 111)
I20260929 03:26:44.147842 18134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:44.150456 18134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:44.150629 18134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:44.154808 18134 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.125:44635
I20260929 03:26:44.154883 18201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.125:44635 every 8 connection(s)
I20260929 03:26:44.155269 18134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/data/info.pb
I20260929 03:26:44.156281 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:44.158010 18202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:44.161970 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 18134
I20260929 03:26:44.162086 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-1/wal/instance
I20260929 03:26:44.162314 27921 external_mini_cluster.cc:1428] Setting key b1b9467462433d975fe2511e411f7ef7
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:44.168236 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.158720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.27.68.126:34177)
0929 03:26:44.158785 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.158787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.159004 (+ 217us) server_negotiation.cc:407] Connection header received
0929 03:26:44.159564 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.159565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.159582 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.159636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.160335 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.160842 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.161858 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.162069 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.162165 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.165076 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.165078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.165078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.165085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.166582 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.167122 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.167123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.167123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.167176 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.167543 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.167544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.167544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.167660 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.167714 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.168095 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.168167 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:26:44.168529 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.158691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34177 (local address 127.0.0.1:58396)
0929 03:26:44.158881 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.158919 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.159447 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.159717 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.159720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.160041 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.160240 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.160245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.160937 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.160938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.161741 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.161743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.162217 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.162219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.162415 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.162963 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.162974 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.164972 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.166669 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.166677 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.166680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.167033 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.167272 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.167272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.167273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.167438 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.167730 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.167737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.167940 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.168073 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.168268 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260929 03:26:44.169163 18202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.68.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.68.124@KRBTEST.COM" created.
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:44.176908 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.169448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59454)
0929 03:26:44.169664 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.169675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.169728 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.169873 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.169874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.170016 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.170068 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.170069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.170973 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.170974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.171621 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.171623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.172114 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.172115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.172179 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.172585 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.172589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.173928 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.175820 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.175821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.175821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.175985 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.176201 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.176201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.176202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.176255 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.176560 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.176561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.176601 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.176683 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.176790 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260929 03:26:44.177201 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.169511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.27.68.125:44635)
0929 03:26:44.169676 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.169681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.169727 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:26:44.169787 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.169789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.169803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.169863 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.170148 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.170876 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.171700 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.172015 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.172115 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.174012 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.174022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.174022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.174031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.175723 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.176079 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.176083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.176083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.176136 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.176333 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.176333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.176334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.176479 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.176566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.176717 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.176866 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260929 03:26:44.177861 18202 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:44.178246 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.178134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45182)
0929 03:26:44.178180 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.178227 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:26:44.178375 18151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
I20260929 03:26:44.178787 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.178700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45192)
0929 03:26:44.178740 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.178764 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:26:44.178897 18202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
Entry for principal kudu/127.27.68.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab.
I20260929 03:26:44.189589 27921 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko51MCN/build/release/bin/kudu
/tmp/dist-test-tasko51MCN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:34943
--webserver_interface=127.27.68.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.68.84:38599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:26:44.199888 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } attempt: 3
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:44.207372 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.200303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59458)
0929 03:26:44.200357 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.200372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.200446 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.200594 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.200595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.200787 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.200875 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.200877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.201531 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.201532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.202402 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.202405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.202765 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.202766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.202858 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.203352 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.203358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.204689 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.206292 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.206294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.206294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.206475 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.206689 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.206690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.206690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.206754 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.206999 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.207000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.207051 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.207134 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.207266 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 03:26:44.207372 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.200373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.27.68.125:44635)
0929 03:26:44.200449 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.200451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.200498 (+ 47us) server_negotiation.cc:407] Connection header received
0929 03:26:44.200524 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.200525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.200534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.200577 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.200954 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.201457 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.202492 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.202700 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.202748 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.204799 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.204801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.204802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.204809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.206192 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.206562 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.206563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.206564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.206622 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.206816 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.206817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.206817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.206932 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.207007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.207160 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.207272 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:26:44.207808 18127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:44.208189 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.208088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45204)
0929 03:26:44.208139 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.208170 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:26:44.208570 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.208478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45210)
0929 03:26:44.208520 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.208543 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:26:44.208659 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
I20260929 03:26:44.228119 18202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 1
I20260929 03:26:44.228627 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.228495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45220)
0929 03:26:44.228552 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.228596 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:26:44.229020 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.228879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45228)
0929 03:26:44.228944 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.228987 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 03:26:44.229156 18202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
I20260929 03:26:44.239799 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 1
I20260929 03:26:44.240279 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.240147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45234)
0929 03:26:44.240210 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.240251 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:26:44.240619 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.240523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45242)
0929 03:26:44.240567 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.240592 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:26:44.240710 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
W20260929 03:26:44.275202 18209 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:26:44.275511 18209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:26:44.275553 18209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:26:44.275592 18209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:26:44.276784 18209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:26:44.276865 18209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:26:44.276888 18209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:26:44.276911 18209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:26:44.276938 18209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:26:44.278575 18209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.68.84:38599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.68.126:34177,127.27.68.125:44635,127.27.68.124:34943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/krb5kdc/kudu_127.27.68.124.keytab
--principal=kudu/127.27.68.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.68.124:34943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.68.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:26:44.278868 18209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:26:44.279111 18209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:26:44.283841 18217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:26:44.283855 18214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:44.283900 18209 server_base.cc:1034] running on GCE node
W20260929 03:26:44.283852 18215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:26:44.284408 18209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:26:44.284679 18209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:26:44.285832 18209 hybrid_clock.cc:648] HybridClock initialized: now 1790652404285819 us; error 31 us; skew 500 ppm
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:26:44.288602 18209 init.cc:377] Logged in from keytab as kudu/127.27.68.124@KRBTEST.COM (short username kudu)
I20260929 03:26:44.289044 18209 webserver.cc:460] Webserver started at http://127.27.68.124:39725/ using document root <none> and password file <none>
I20260929 03:26:44.289264 18209 fs_manager.cc:359] Metadata directory not provided
I20260929 03:26:44.289310 18209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:26:44.289418 18209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:26:44.290498 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data/instance:
uuid: "f7c8c7b210a34eabae62935b4401ccb6"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "4287290ecd10428119b2654308a7a42c"
server_key_iv: "67c8315a63b2210f749dc2a20185e550"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.290823 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal/instance:
uuid: "f7c8c7b210a34eabae62935b4401ccb6"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "4287290ecd10428119b2654308a7a42c"
server_key_iv: "67c8315a63b2210f749dc2a20185e550"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.292413 18209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:26:44.293172 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:44.293350 18209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:26:44.293443 18209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data,/tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal
uuid: "f7c8c7b210a34eabae62935b4401ccb6"
format_stamp: "Formatted at 2026-09-29 03:26:44 on dist-test-slave-fzwk"
server_key: "4287290ecd10428119b2654308a7a42c"
server_key_iv: "67c8315a63b2210f749dc2a20185e550"
server_key_version: "encryptionkey@0"
I20260929 03:26:44.293509 18209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:26:44.297288 18202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 2
I20260929 03:26:44.297706 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.297580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45250)
0929 03:26:44.297637 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.297672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:26:44.298017 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.297923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45262)
0929 03:26:44.297967 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.297993 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 03:26:44.298116 18202 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
I20260929 03:26:44.314723 18209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:26:44.315841 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 2
I20260929 03:26:44.316277 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.316132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45268)
0929 03:26:44.316195 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.316239 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:26:44.317044 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.316916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45278)
0929 03:26:44.316977 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.317010 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:26:44.317169 18127 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.68.124:34943: Network error: Client connection negotiation failed: client connection to 127.27.68.124:34943: connect: Connection refused (error 111)
I20260929 03:26:44.317379 18209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:26:44.317489 18209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:26:44.321496 18209 rpc_server.cc:307] RPC server started. Bound to: 127.27.68.124:34943
I20260929 03:26:44.321563 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.68.124:34943 every 8 connection(s)
I20260929 03:26:44.321924 18209 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/data/info.pb
I20260929 03:26:44.322943 18277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:26:44.324846 27921 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko51MCN/build/release/bin/kudu as pid 18209
I20260929 03:26:44.324630 18277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:44.324954 27921 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko51MCN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652132259900-27921-0/minicluster-data/master-2/wal/instance
I20260929 03:26:44.325141 27921 external_mini_cluster.cc:1428] Setting key 68ad0324e73a68ab33984f69228d8e06
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:44.334836 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.325518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.27.68.126:34177)
0929 03:26:44.325587 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.325589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.325880 (+ 291us) server_negotiation.cc:407] Connection header received
0929 03:26:44.326439 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.326440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.326453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.326500 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.327047 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.327598 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.328640 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.328939 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.329012 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.331553 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.331556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.331556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.331564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.333192 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.333717 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.333718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.333719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.333790 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.334074 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.334076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.334076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.334245 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.334322 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.334648 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.334741 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:26:44.335086 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.325520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.126:34177 (local address 127.0.0.1:58400)
0929 03:26:44.325750 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.325792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.326321 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.326528 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.326531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.326797 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.326958 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.326962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.327674 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.327675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.328566 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.328568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.329065 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.329066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.329200 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.329786 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.329795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.331455 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.333279 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.333289 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.333292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.333633 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.333872 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.333873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.333873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.334002 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.334319 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.334325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.334495 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.334616 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.334805 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
I20260929 03:26:44.335626 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.325591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.27.68.124:34943)
0929 03:26:44.325788 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.325798 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.325832 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:26:44.326325 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.326334 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.326363 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.326456 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.327086 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.328180 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.329068 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.329511 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.329568 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.332193 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.332201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.332201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.332208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.334087 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.334457 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.334460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.334461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.334519 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.334742 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.334743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.334743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.334908 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.335057 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.335372 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.335461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
I20260929 03:26:44.335654 18277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.126@KRBTEST.COM
I20260929 03:26:44.343144 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.335959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.125:44635 (local address 127.0.0.1:59462)
0929 03:26:44.336002 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.336013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.336059 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.336289 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.336290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.336450 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.336507 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.336509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.337405 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.337406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.338116 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.338118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.338581 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.338582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.338634 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.339043 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.339049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.340290 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.342100 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.342100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.342101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.342324 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.342518 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.342519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.342519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.342596 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.342865 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.342866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.342920 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.342991 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.343076 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:26:44.343292 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.336034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.27.68.125:44635)
0929 03:26:44.336086 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.336088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.336097 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:26:44.336126 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.336127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.336135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.336185 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.336589 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.337335 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.338189 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.338496 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.338580 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.340379 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.340381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.340381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.340390 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.342007 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.342396 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.342397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.342397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.342450 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.342674 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.342675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.342675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.342801 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.342854 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.343036 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.343129 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:26:44.343639 18277 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:26:44.344204 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.336443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.27.68.126:34177)
0929 03:26:44.336488 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.336490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.336531 (+ 41us) server_negotiation.cc:407] Connection header received
0929 03:26:44.336606 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.336606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.336615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.336663 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.337213 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.337872 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.338757 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.339016 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.339084 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.341298 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.341300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.341300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.341307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.342866 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.343276 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.343277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.343278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.343337 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.343633 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.343634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.343634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.343788 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.343966 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.344011 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.344105 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.124@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.68.125@KRBTEST.COM
I20260929 03:26:44.351115 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.343906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45288)
0929 03:26:44.343952 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.343962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.343994 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.344129 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.344130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.344419 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.344463 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.344465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.345216 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.345217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.345892 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.345893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.346345 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.346346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.346397 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.346767 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.346773 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.348233 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.349957 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.349958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.349958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.350165 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.350409 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.350410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.350410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.350474 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.350726 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.350727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.350780 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.350852 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.350963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:26:44.351116 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.343959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.27.68.124:34943)
0929 03:26:44.343996 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.343998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.344005 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:26:44.344032 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.344032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.344053 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.344113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.344564 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.345143 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.345960 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.346281 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.346340 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.348341 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.348342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.348343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.348349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.349876 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.350248 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.350249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.350250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.350314 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.350540 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.350541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.350541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.350660 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.350714 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.350892 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.351030 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":20}
I20260929 03:26:44.352952 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.344848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.27.68.125:44635)
0929 03:26:44.344883 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.344885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.344934 (+ 49us) server_negotiation.cc:407] Connection header received
0929 03:26:44.345004 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.345005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.345014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.345067 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.345658 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.346201 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.347125 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.347409 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.347484 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.350035 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.350037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.350037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.350044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.351870 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.352270 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.352271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.352271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.352319 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.352509 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.352509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.352510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.352614 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.352732 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.352832 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.352890 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:26:44.353051 18277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: Bootstrap starting.
I20260929 03:26:44.354158 18277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:44.354422 27921 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:26:44.354526 18277 log.cc:824] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:44.355510 18277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: No bootstrap required, opened a new log
I20260929 03:26:44.355477 18101 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58404
I20260929 03:26:44.357594 18277 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.357764 18277 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8c7b210a34eabae62935b4401ccb6, State: Initialized, Role: FOLLOWER
I20260929 03:26:44.357843 18277 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.358199 18277 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:44.358176 18283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.358259 18283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:26:44.360078 18294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:44.360145 18294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:44.360450 18209 master_runner.cc:186] Error getting master registration for 127.27.68.126:34177: OK, instance_id { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" instance_seqno: 1790652403972768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:26:44.362522 18209 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:26:44.389359 18202 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 3
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652404, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.125@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:44.397840 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.389873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.27.68.124:34943)
0929 03:26:44.389941 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.389943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.389961 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:26:44.390063 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.390064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.390075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.390130 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.390576 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.391363 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.392385 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.392696 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.392767 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.394933 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.394934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.394935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.394943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.396577 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.396964 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.396966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.396966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.397025 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.397207 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.397208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.397209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.397390 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.397468 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.397646 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.397741 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:44.397840 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.389798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45304)
0929 03:26:44.389894 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.389908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.389973 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.390180 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.390180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.390418 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.390483 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.390485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.391446 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.391448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.392283 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.392285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.392788 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.392789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.392849 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.393362 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.393368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.394839 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.396658 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.396660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.396660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.396876 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.397080 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.397081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.397081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.397138 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.397460 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.397460 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.397510 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.397617 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.397741 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 03:26:44.400233 18202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: Bootstrap starting.
I20260929 03:26:44.400906 18202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:44.401237 18202 log.cc:824] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:44.401994 18202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: No bootstrap required, opened a new log
I20260929 03:26:44.403534 18202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.403682 18202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8e09ff2f8c41bbadf6f25c7b04cf13, State: Initialized, Role: FOLLOWER
I20260929 03:26:44.403777 18202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.404078 18297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.404160 18202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:44.404173 18297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:26:44.405995 18308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:44.406050 18308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:44.406286 18134 master_runner.cc:186] Error getting master registration for 127.27.68.126:34177: OK, instance_id { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" instance_seqno: 1790652403972768 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:26:44.408627 18134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:26:44.416321 18127 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } attempt: 3
Sep 29 03:26:44 dist-test-slave-fzwk krb5kdc[18046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652403, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.68.126@KRBTEST.COM for kudu/127.27.68.124@KRBTEST.COM
I20260929 03:26:44.428105 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.416674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.68.124:34943 (local address 127.0.0.1:45312)
0929 03:26:44.416735 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 03:26:44.416747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:26:44.416788 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:26:44.416956 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:26:44.416957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:26:44.417166 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:26:44.420186 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.420194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.421211 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.421213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.422068 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:26:44.422071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.422644 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.422646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:26:44.422711 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.423364 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:26:44.423370 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:26:44.424976 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:26:44.426921 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.426924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.426924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.427169 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.427417 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:26:44.427418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:26:44.427419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:26:44.427486 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:26:44.427761 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:26:44.427761 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:26:44.427814 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 03:26:44.427875 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:26:44.427986 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:26:44.428128 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.416764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.27.68.124:34943)
0929 03:26:44.416810 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.416812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.416823 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:26:44.416852 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.416853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.416863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.416940 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.420326 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.421120 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.422179 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.422494 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.422571 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.425083 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.425085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.425085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.425094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.426826 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.427281 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.427282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.427283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.427348 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.427558 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.427559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.427559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.427705 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.427773 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.427938 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.428021 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:26:44.430320 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: Bootstrap starting.
I20260929 03:26:44.431142 18127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: Neither blocks nor log segments found. Creating new log.
I20260929 03:26:44.431510 18127 log.cc:824] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:26:44.432493 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: No bootstrap required, opened a new log
I20260929 03:26:44.433965 18127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.434147 18127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c58cb9bb0b4c11ab9603e42f784f9b, State: Initialized, Role: FOLLOWER
I20260929 03:26:44.434258 18127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.434608 18127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:26:44.434631 18101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:26:44.434705 18101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:44.435702 18101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.436372 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13"
I20260929 03:26:44.436514 18176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:26:44.436932 18322 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:26:44.436980 18322 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:26:44.437737 18176 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c58cb9bb0b4c11ab9603e42f784f9b in term 1.
I20260929 03:26:44.438406 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f7c8c7b210a34eabae62935b4401ccb6"
I20260929 03:26:44.438531 18251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:26:44.439491 18251 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c58cb9bb0b4c11ab9603e42f784f9b in term 1.
I20260929 03:26:44.439779 18076 leader_election.cc:302] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c58cb9bb0b4c11ab9603e42f784f9b, f7c8c7b210a34eabae62935b4401ccb6; no voters:
I20260929 03:26:44.435978 18101 leader_election.cc:288] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [CANDIDATE]: Term 1 election: Requested vote from peers 6f8e09ff2f8c41bbadf6f25c7b04cf13 (127.27.68.125:44635), f7c8c7b210a34eabae62935b4401ccb6 (127.27.68.124:34943)
I20260929 03:26:44.443293 18324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:26:44.443475 18324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [term 1 LEADER]: Becoming Leader. State: Replica: 92c58cb9bb0b4c11ab9603e42f784f9b, State: Running, Role: LEADER
I20260929 03:26:44.443650 18059 master_runner.cc:263] No leader master found from master 92c58cb9bb0b4c11ab9603e42f784f9b
I20260929 03:26:44.443603 18324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } }
I20260929 03:26:44.443688 18059 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:26:44.443811 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.443939 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:44.444092 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c58cb9bb0b4c11ab9603e42f784f9b. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.444212 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:44.445214 18331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:26:44.445451 18331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:26:44.458962 18251 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [term 1 FOLLOWER]: Refusing update from remote peer 92c58cb9bb0b4c11ab9603e42f784f9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:44.458962 18176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [term 1 FOLLOWER]: Refusing update from remote peer 92c58cb9bb0b4c11ab9603e42f784f9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:26:44.459376 18329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:44.459645 18311 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:26:44.461462 18297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c58cb9bb0b4c11ab9603e42f784f9b. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.461602 18297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:44.463480 18283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92c58cb9bb0b4c11ab9603e42f784f9b. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.463594 18283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:44.465155 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.444583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.27.68.125:44635)
0929 03:26:44.444637 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.444640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.446553 (+ 1913us) server_negotiation.cc:407] Connection header received
0929 03:26:44.446655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.446656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.446667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.446698 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.448459 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.449212 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.452029 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.452339 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.452388 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.456129 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.456130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.456131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.456139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.457958 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.458408 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.458409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.458410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.458471 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.464077 (+ 5606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.464079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.464080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.464255 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.464345 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.464939 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.465035 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:26:44.468216 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.446369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.27.68.124:34943)
0929 03:26:44.446418 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.446420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.459971 (+ 13551us) server_negotiation.cc:407] Connection header received
0929 03:26:44.460065 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.460066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.460078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.460109 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.460819 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.461506 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.462591 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.462851 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.462900 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.464490 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.464491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.464492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.464498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.466772 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.467274 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.467275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.467275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.467350 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.467574 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.467575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.467575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.467722 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.467793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.468033 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.468111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:26:44.468709 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.468808 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:44.468906 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:44.444962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.27.68.126:34177)
0929 03:26:44.444999 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.445001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.447598 (+ 2597us) server_negotiation.cc:407] Connection header received
0929 03:26:44.447686 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.447687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.447697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.447725 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.450245 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.450942 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.454072 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:44.454368 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:44.454412 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:44.464417 (+ 10005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:26:44.464418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:26:44.464418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:26:44.464427 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:26:44.466781 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.467752 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.467753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.467753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.467806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:26:44.468035 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:26:44.468036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:26:44.468037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:26:44.468163 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:26:44.468265 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:44.468687 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 03:26:44.468770 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:26:44.469395 18283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.469473 18283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:44.469545 18338 mvcc.cc:205] Tried to move back new op lower bound from 7334512248661147648 to 7334512248601903104. Current Snapshot: MvccSnapshot[applied={T|T < 7334512248661147648}]
I20260929 03:26:44.471261 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.471355 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b [sys.catalog]: This master's current role is: LEADER
I20260929 03:26:44.471942 18297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c58cb9bb0b4c11ab9603e42f784f9b" member_type: VOTER last_known_addr { host: "127.27.68.126" port: 34177 } } peers { permanent_uuid: "6f8e09ff2f8c41bbadf6f25c7b04cf13" member_type: VOTER last_known_addr { host: "127.27.68.125" port: 44635 } } peers { permanent_uuid: "f7c8c7b210a34eabae62935b4401ccb6" member_type: VOTER last_known_addr { host: "127.27.68.124" port: 34943 } } }
I20260929 03:26:44.472038 18297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:26:44.472540 18336 mvcc.cc:205] Tried to move back new op lower bound from 7334512248661147648 to 7334512248601903104. Current Snapshot: MvccSnapshot[applied={T|T < 7334512248661147648}]
I20260929 03:26:44.475768 18331 catalog_manager.cc:1347] Generated new cluster ID: a3ebb3afbed54179bee70f3e32525894
I20260929 03:26:44.475837 18331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:26:44.496193 18331 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:26:44.498365 18331 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:26:44.506055 18331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92c58cb9bb0b4c11ab9603e42f784f9b: Generated new TSK 0
I20260929 03:26:44.506381 18331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:26:44.548111 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.546261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.27.68.125:44635)
0929 03:26:44.546309 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.546311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.547399 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 03:26:44.547517 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.547518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.547530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.547562 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.548036 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59498: BlockingRecv error: recv got EOF from 127.0.0.1:59498 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:26:44.549276 18130 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.546730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.27.68.126:34177)
0929 03:26:44.546775 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.546777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.548551 (+ 1774us) server_negotiation.cc:407] Connection header received
0929 03:26:44.548652 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.548653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.548665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.548693 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.549202 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58430: BlockingRecv error: recv got EOF from 127.0.0.1:58430 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:26:44.550209 18281 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.547070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.27.68.124:34943)
0929 03:26:44.547110 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.547112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.548926 (+ 1814us) server_negotiation.cc:407] Connection header received
0929 03:26:44.549638 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.549639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.549650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.549680 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.550149 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45318: BlockingRecv error: recv got EOF from 127.0.0.1:45318 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:26:44.553292 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.552336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.27.68.125:44635)
0929 03:26:44.552380 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.552382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.552699 (+ 317us) server_negotiation.cc:407] Connection header received
0929 03:26:44.552814 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.552816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.552836 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.552865 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.553238 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59500: BlockingRecv error: recv got EOF from 127.0.0.1:59500 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:26:44.556813 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.555851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.27.68.125:44635)
0929 03:26:44.555904 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.555906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.556252 (+ 346us) server_negotiation.cc:407] Connection header received
0929 03:26:44.556359 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.556360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.556375 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.556422 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.556756 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59502: BlockingRecv error: recv got EOF from 127.0.0.1:59502 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:26:44.560971 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.560044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.27.68.125:44635)
0929 03:26:44.560083 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.560085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.560300 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:26:44.560413 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.560414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.560427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.560457 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.560916 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59504: BlockingRecv error: recv got EOF from 127.0.0.1:59504 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 03:26:44.567370 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.566468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.27.68.125:44635)
0929 03:26:44.566509 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.566511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.566752 (+ 241us) server_negotiation.cc:407] Connection header received
0929 03:26:44.566854 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.566855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.566867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.566912 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.567307 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59516: BlockingRecv error: recv got EOF from 127.0.0.1:59516 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:26:44.577471 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.576744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.27.68.125:44635)
0929 03:26:44.576792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.576794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.576914 (+ 120us) server_negotiation.cc:407] Connection header received
0929 03:26:44.576989 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.576990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.577002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.577079 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.577402 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59518: BlockingRecv error: recv got EOF from 127.0.0.1:59518 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:26:44.595467 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.594602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.27.68.125:44635)
0929 03:26:44.594670 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.594672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.594878 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:26:44.594954 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.594955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.594967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.595017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.595406 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59530: BlockingRecv error: recv got EOF from 127.0.0.1:59530 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260929 03:26:44.629179 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.628521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59532 (local address 127.27.68.125:44635)
0929 03:26:44.628584 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.628586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.628667 (+ 81us) server_negotiation.cc:407] Connection header received
0929 03:26:44.628702 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.628703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.628716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.628797 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.629118 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59532: BlockingRecv error: recv got EOF from 127.0.0.1:59532 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 03:26:44.695178 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.694460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.27.68.125:44635)
0929 03:26:44.694518 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.694520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.694593 (+ 73us) server_negotiation.cc:407] Connection header received
0929 03:26:44.694671 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.694671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.694682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.694723 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.695098 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59538: BlockingRecv error: recv got EOF from 127.0.0.1:59538 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 03:26:44.825058 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:44.824360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.27.68.125:44635)
0929 03:26:44.824456 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:44.824459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:44.824486 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:26:44.824566 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:44.824567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:44.824579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:44.824635 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:44.824990 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59548: BlockingRecv error: recv got EOF from 127.0.0.1:59548 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 03:26:45.083169 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:26:45.082410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.27.68.125:44635)
0929 03:26:45.082477 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.082479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.082503 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:26:45.082626 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.082627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.082644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.082708 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:26:45.083094 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59552: BlockingRecv error: recv got EOF from 127.0.0.1:59552 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:26:45.360599 18294 catalog_manager.cc:1259] Loaded cluster ID: a3ebb3afbed54179bee70f3e32525894
I20260929 03:26:45.360675 18294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: loading cluster ID for follower catalog manager: success
I20260929 03:26:45.361497 18294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: acquiring CA information for follower catalog manager: success
I20260929 03:26:45.361812 18294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7c8c7b210a34eabae62935b4401ccb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:26:45.406499 18308 catalog_manager.cc:1259] Loaded cluster ID: a3ebb3afbed54179bee70f3e32525894
I20260929 03:26:45.406587 18308 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: loading cluster ID for follower catalog manager: success
I20260929 03:26:45.407394 18308 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: acquiring CA information for follower catalog manager: success
I20260929 03:26:45.407655 18308 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f8e09ff2f8c41bbadf6f25c7b04cf13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:26:45.599056 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:45.596446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.27.68.125:44635)
0929 03:26:45.596564 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.596566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.596591 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:26:45.596684 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.596685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.596735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.596820 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:45.597003 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.597552 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.598300 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.598491 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.598546 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:45.598687 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:45.598798 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:45.598859 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:45.598893 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 03:26:45.598917 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260929 03:26:45.603027 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:45.600873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.27.68.124:34943)
0929 03:26:45.600974 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.600976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.600984 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:26:45.601041 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.601042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.601078 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.601148 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:45.601229 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.601701 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.602384 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.602584 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.602634 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:45.602756 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:45.602860 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:45.602912 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:45.602945 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:26:45.602961 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":49,"threads_started":1}
I20260929 03:26:45.607674 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:45.604695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.27.68.125:44635)
0929 03:26:45.604949 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.604951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.605086 (+ 135us) server_negotiation.cc:407] Connection header received
0929 03:26:45.605121 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.605122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.605134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.605185 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:45.605331 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.605828 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.606851 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.607139 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.607207 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:45.607366 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:45.607399 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:45.607448 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:45.607567 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:26:45.607596 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260929 03:26:45.608119 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:45.604943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.27.68.124:34943)
0929 03:26:45.604982 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.604984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.604990 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:26:45.605012 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.605012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.605022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.605101 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:45.605386 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.606135 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.607153 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.607477 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.607547 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:45.607831 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:45.607867 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:45.607896 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:45.607992 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:26:45.608022 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:26:45.608155 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:26:45.604703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.27.68.126:34177)
0929 03:26:45.604867 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:26:45.604869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:26:45.605043 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:26:45.605093 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:26:45.605094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:26:45.605144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:26:45.605246 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:26:45.605418 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.606197 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.606989 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:26:45.607342 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:26:45.607410 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:26:45.607584 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:26:45.607694 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:26:45.607765 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:26:45.608016 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 03:26:45.608058 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260929 03:26:45.609881 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 18059
I20260929 03:26:45.615477 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 18134
I20260929 03:26:45.620450 27921 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko51MCN/build/release/bin/kudu with pid 18209
2026-09-29T03:26:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2940 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5726 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273366 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST